![]() In addition, you want to make the remaining data transfers as efficient as possible:ġ. You need to minimize costly data transfers between the application and the database through what is known as in-database analytics or in-database processing. ![]() This requires optimizing the division of labor between the application and the underlying database. When you have a high-performance database, you also need to use it effectively. Let the Database Do the Heavy LiftingĬoming back to the race car metaphor, columnar databases, effective compression, and in-memory data processing correspond to the race car’s V8 engine, but it takes more than a powerful engine to make your car win races. This obstacle can be overcome by smart engineering, such as the use of sparse indexes for sorted data. However, a nice illustration of the impact of seemingly innocuous technical choices is that traditional indexing approaches for speedier location of data can result in index tables that are a thousand times larger than the actual compressed database. For example, several years of cleansed sales and weather data per store, product and day are needed to predict the impact of forecasted weather conditions on sales.ĭata compression is a science of its own, and we will not go into details here. In retail planning and analytics, this is very common. As sequential memory access is significantly quicker than fetching data from multiple locations in physical memory, columnar databases deliver superior performance when a lot of data stored in a limited number of columns needs to be processed together. Accessing data in memory still introduces latency, although it is much smaller than when accessing data on disk. For example, a column consisting of retail sales data would typically contain a lot of small integer values and repetitive zeros.Ģ. Column-store databases typically contain long sequences of similar or even repetitive data that allow data to be compressed far more efficiently than when using row-store databases. Efficient compression is a must to enable in-memory data processing of large amounts of data. Amazingly Fast Analytical Data ProcessingĬonsidering that fetching data from disk can be a thousand times slower than fetching data from memory, it is no surprise that in-memory computing (IMC) has become the norm for data-intensive applications, especially when on-demand analytics are involved.Ĭolumnar databases commonly go together with in-memory data processing, for two main reasons:ġ. Smart optimization in these areas allows for performance improvements of several orders of magnitude. Rather, performance is more often determined by how quickly data can be found in the data storage and fetched from the data storage to the central processing unit (CPU), as well as how efficiently calculation results can be transferred from the underlying database to the application layer. Pushing the Limits of Data Processing PerformanceĪ surprising fact when dealing with data-intensive applications, such as retail planning and analytics software, is that the actual computations seldom form the performance bottleneck. ![]() When pushing the limits of high-performance data processing, you need to specialize to win the race. To summarize, the biggest differences between a NASCAR race car and the car in your driveway boil down to specialization. If you did that to a street car, the engine would not work well at lower, street-legal speeds.Īdditionally, to turn all this power into maximum speed, race cars have been designed for optimal aerodynamics and downforce, minimum weight with sufficient robustness, and favorable weight distribution. In a race car, the cam keeps the intake valves open longer at certain higher speeds for additional airflow and peak engine performance. The cam regulates the amount of fuel and air mixture the engine can pull in and push out, which in turn dictates the power the engine will generate. The V8 engine is about four times more powerful than most four-cylinder engines on street cars, but another important difference is to be found in how the engine’s camshaft works. In fact, the engine of a NASCAR car is not really that different from a street car’s. You obviously cannot win the NASCAR Cup with a street car, not even with one that has a very big and expensive engine. When pushing the limits of performance, specialization is a must.
0 Comments
Leave a Reply. |