Processing Modes of Data Transfer Processes 

There are various processing modes for processing a data transfer process request (DTP request) with substep extraction and processing (transformation and update).

Background Processing Modes for Standard Data Transfer Processes

The request of a standard DTP should always be processed in as many parallel processes as possible. There are 3 processing modes for background processing of standard DTPs. Each processing mode stands for a different degree of parallelization:

●      Parallel extraction and processing (transformation and update)

The data packages are extracted and processed in parallel process, meaning that a parallel process is derived from the main process for each data package. This parallel process extracts and processes the data.

You can define the maximum number of background processes that can be used for each DTP.

This processing method is referred to below as processing mode 1 (P1).

●      Serial extraction, immediate parallel processing

The data packages are extracted sequentially in a process. The packages are processed in parallel processes, meaning that the main process extracts the data packages sequentially and derives a process that processes the data for each data package.

You can define the maximum number of background processes that can be used for each DTP.

This processing method is referred to below as processing mode 2 (P2).

●      Serial extraction and processing of the source packages

The data packages are extracted and processed sequentially in a process, the main process.

This processing method is referred to below as processing mode 3 (P3).

Processing mode 1 offers the best performance, while processing mode 3 offers the lowest level of performance. The choice of processing mode for a given DTP (as a combination of source, transformation and target) depends on the properties of the extractor, the transformation, and the target.

Criteria for Selecting the Processing Mode

●        Semantic Grouping Possible

An extractor has this property if it can return data for a grouping key defined in the DTP package by package to the caller as a semantic unit. The semantic grouping is possible for the following sources: DataSource, DataStore object and InfoCube.

Grouping Key and Grouping Mode

Grouping Key

The grouping key is the subset of the source fields defined in the DTP for the semantic grouping (tab page Extraction → pushbutton Semantic Groups). It defines how the data packages that are read from the source (DataSource, DataStore object or InfoCube) are created. The data records for a grouping key are combined into one data package. The grouping key is also the key for the error stack of the DTP.  

The grouping key for the source depends on whether error handling is activated for the DTP and whether the transformations called within the DTP and the target require semantically grouped data:

●      Depending on error handling

If error handling is activated, grouping is required in order to define the key fields for the error stack. This is relevant for DataStore objects with data fields that are overwritten. The target key represents the error stack key for targets in which the order of the updated data is of no importance (such as additive delta in InfoCubes); it is marked as the grouping key in the DTP. 

●      Depending on transformation and target

The example below shows how the transformation and target of a DTP influence the grouping key:

Update from a DataSource that can provide the stock prices accurately to the minute into a DataStore object in which the prices at the end of the day are kept for a given security identity number.

In this example, the transformation between the DataSource and the DataStore object has the task of copying the last stock price of the day to the target and filtering out all other prices. To do this, all values for a given security identity number and date are provided for the exact minute in a package. The grouping key here would be the security identity number and the calendar date.

Grouping Modes

The grouping mode defines whether a semantic grouping is required and whether a grouping key exists in the DTP. As explained, grouping is required if error handling is activated. The following grouping modes are possible:

●      Case 1: No grouping is required; the grouping key includes all the fields of the source.

●      Case 2: Grouping is required. There is a grouping key that does not include all the fields of the source.

●      Case 3: Grouping is required. The grouping key does not contain any fields. This corresponds to an empty set.

Packaged Data in the Source

The data in the source is already available in standardized package form. This is supported by the DataSource source.

Order when Updating to the Target is of no Relevance (Commutative Update)

The data is stored in the target so that it can always be updated in parallel. Grouping is not even required if the transformation asks for grouping.

Decision Tree for Defining the Processing Mode

The figure below illustrates how the system defines one of the described processing modes based on the system properties described above:

 

Other Processing Modes

The DTP provides further processing modes for special applications and access methods:

Serial in the dialog process (for debugging)

With this processing mode you execute the data transfer process in debugging mode. The request is processed synchronously in a dialog process and the update of the data is simulated.

No data transfer; delta status in source: fetched

With this processing mode you execute a delta without transferring data. This is analogous to simulating the delta initialization with the InfoPackage. In this case you execute the DTP directly in the dialog.

Processing mode for real-time data packages

With this processing mode you execute data transfer processes for real-time data acquisition.

Processing mode for direct access

With this processing mode you execute data transfer processes for direct access.