Technical Information About the HPA Engine  

Processing Data

After creating a HPA index, the data is available on the file server of the HPA server. The data is loaded to the main memory when you execute a query for the first time or start a special load program. The data remains in the main memory until it is replaced or is removed from the main memory when a special delete program is started. It may be necessary to execute a special delete program if, for example, there is not enough memory on the HPA server for all HPA indexes and you need to load data from particular InfoCubes but data from other InfoCubes is not needed (at this time).

Table data is stored in the main memory in columns. Vertically segmenting data tables in this way is more efficient than saving row-based data in conventional relational database systems. In a conventional database, the system has to search all the data in the table if a predefined aggregate is not available for a query.  The HPA engine specifically accesses only those data columns that are relevant. It sorts the columns individually and puts the required entry at the beginning. This improves performance considerably because the data flows are smaller. It also significantly reduces the input and output load and the main memory consumption.

Compressing Data

Data is available on the HPA server in a read-optimized format. The HPA engine uses dictionary-based compression. Integers are used to represent text or values in table cells. Using integers allows efficient numeric coding and intelligent caching strategies.

For example, if a column has a thousand rows and some of the cells contain long texts, efficiency is significantly increased by using a ten-bit binary number to identify the texts during processing and a dictionary to call them again afterwards. The datasets that have to be transferred and temporarily stored during the different processing steps are reduced on average by a factor of ten.

This means that you can perform the entire query processing in the main memory and reduces network traffic between separate landscapes.

Divided (Split) Indexes

The HPA engine can process huge datasets, without exceeding the limits of the installed memory architecture. You can split large tables (fact tables and large X and Y tables) horizontally, save them on different PCs and process them quickly in parallel. The maximum size a table can be before the system splits the index, depends on the existing hardware of the HPA server. Data is distributed to the sub-indexes in a round-robin procedure. Write, optimize and read accesses are parallelized on the HPA server.

This scalability allows users to make use of sophisticated adaptive computing infrastructures such as blade servers and grid computing.