Packaged Data Transfer from an SAP HANA Source System

Background

A data transfer process generally divides the data in a request into multiple packages, to ensure that transformation (and further processing) is carried out in a resource-efficient manner. In particular, this ensures that sufficient system resources are available for transformation (and further processing) (for example, there is no memory overflow). You configure the package size in the data transfer process.

Special Feature of SAP HANA Source Systems

A data transfer process that reads data from an SAP HANA source system uses disjunctive selections on specific columns to create package of the configured size. You configure which columns are used as a property of the fields of a DataSource. Choose columns for packaging and partitioning that have neither a particularly high nor a particularly low cardinality. If, for example, approximately 10,000 records are added to a source object each month, the calendar month (including year) would be a suitable column for packaging and partitioning. If you select multiple columns, disjunctive selections are made across all columns selected. In this case, you should ensure that the correlation between the selected columns is as low as possible.

Special Feature of Transformation in SAP HANA

If a data transfer process divides the data from an SAP HANA source system into multiple packages, there are important differences between transformation in ABAP and in SAP HANA. In the case of transformation in ABAP, the packages are read using a database cursor. All packages of a request are therefore extracted in the same read transaction. Consequently, the read consistency corresponds to the "Read Committed" isolation level of the SAP HANA database. In the case of transformation in SAP HANA, the packages are extracted in different processes and, therefore, in independent read transactions. As a result, the read consistency corresponds to the "Read Uncommitted" isolation level. If write transactions are completed on the read data between the read transactions of the DTP request, "dirty reads" can occur. This can result in a difference between the result of a transformation in SAP HANA and the result that would have been delivered by the same transformation in ABAP.

Procedure for Avoiding Inconsistencies When Reading Packaged Data

If you have sufficient system resources in a specific case, you can configure the DTP to process all of the data in one package. If you want or require multiple packages, you can avoid the inconsistencies described by ensuring that the data set selected does not change during the independent read transactions. It is important to differentiate between the following cases:

1. Reading from a local target table for real-time replication (remote subscription type UPSERT or INSERT): In this case, the system automatically ensures consistent read behavior using the strictly monotonously increasing modification timestamp. Note that this option is available for some data provisioning adapters only.

2. Reading of initial data or data changes (generic BW delta) from a source object: In this case, you can ensure that the selected data set does not change during the independent read transactions by specifying an appropriate upper safety interval. We also recommend that you use the delta-relevant field (numeric pointer, timestamp, or date) for packaging and partitioning. This ensures that the last records changed are processed in the last package.

3. Reading of the current data (full) from a local database object: In this case, there is no generally valid approach. If you cannot ensure that the selected data set does not change by means of appropriate administrative measures (for example, locks or time), check whether the transformation can be executed in ABAP or the "Read Committed" isolation level can be dispensed with.

Note that in the second and third case, it is irrelevant whether your SAP HANA source system points to a local database schema or to a source (remote source using SAP HANA smart data access).