Select language:

Processing Modes in the Data Transfer Process

There are various processing modes for processing a data transfer process request (DTP request) with substep extraction and processing (transformation and update).

Background Processing Modes for Standard Data Transfer Processes

The request of a standard DTP should always be processed in as many parallel processes as possible. There are 3 processing modes for background processing of standard DTPs. Each processing mode stands for a different degree of parallelization:

  • Parallel extraction and processing (transformation and update)

    The data packages are extracted and processed in parallel processes, meaning that a parallel process is derived from the main process for each data package. This parallel process extracts and processes the data.

    Note

    You can define the maximum number of background processes that can be used for each DTP.

    This processing mode is referred to below as processing mode 1 (V1).

    The following sources support parallel extraction: PSA table of the DataSource, write-optimized DataStore object and change log of the standard DataStore object.

  • Serial extraction, immediate parallel processing

    The data packages are extracted sequentially in a process. The packages are processed in parallel processes, meaning that the main process extracts the data packages sequentially and derives a process that processes the data for each data package.

    Note

    You can define the maximum number of background processes that can be used for each DTP.

    This processing mode is referred to below as processing mode 2 (V2).

  • Serial extraction and processing of the source packages

    The data packages are extracted and processed sequentially in a process, the main process.

    This processing mode is referred to below as processing mode 3 (V3).

Processing mode 1 offers the best performance, while processing mode 3 offers the lowest level of performance. The choice of processing mode for a given DTP (as a combination of source, transformation and target) depends on the properties of the extractor, the transformation, and the target.

Criteria for Selecting the Processing Mode

Semantic Grouping Possible

An extractor has this property if it can return data for a grouping key defined in the DTP package by package to the caller as a semantic unit. The semantic grouping is possible for the following sources: DataSource, DataStore object and InfoCube.

Grouping Key and Grouping Mode

  • Grouping Key

    The grouping key is the subset of the source fields defined in the DTP for the semantic grouping (tab page Start of the navigation path Extraktion Next navigation step Semantic Groups End of the navigation path). It defines how the data packages that are read from the source (DataSource, DataStore object or InfoCube) are created. The data records for a grouping key are combined into one data package. The grouping key is also the key for the error stack of the DTP.

    The grouping key for the source depends on whether error handling is activated for the DTP and whether the transformations called within the DTP and the target require semantically grouped data:

    • Depending on error handling

      If error handling is activated, grouping is required in order to define the key fields for the error stack. This is relevant for DataStore objects with data fields that are overwritten. The target key represents the error stack key for targets in which the order of the updated data is of no importance (such as additive delta in InfoCubes); it is marked as the grouping key in the DTP.

    • Depending on transformation and target
      Tip

      The example below shows how the transformation and target of a DTP influence the grouping key:

      Update from a DataSource that can provide the stock prices accurately to the minute into a DataStore object in which the prices at the end of the day are kept for a given security identity number.

      In this example, the transformation between the DataSource and the DataStore object has the task of copying the last stock price of the day to the target and filtering out all other prices. To do this, all values for a given security identity number and date are provided for the exact minute in a package. The grouping key here would be the security identity number and the calendar date.

  • Grouping Modes

    The grouping mode defines whether a semantic grouping is required and whether a grouping key exists in the DTP. As explained, grouping is required if error handling is activated. The following grouping modes are possible:

    • Case 1: No grouping is required; the grouping key includes all the fields of the source.
    • Case 2: Grouping is required. There is a grouping key that does not include all the fields of the source.
    • Case 3: Grouping is required. The grouping key does not contain any fields. This corresponds to an empty set.

Packaged Data in the Source

The data in the source is already available in standardized package form. This is supported by the DataSource source.

Order when Updating to the Target is of no Relevance (Commutative Update)

The data is stored in the target so that it can always be updated in parallel. Grouping is not even required if the transformation asks for grouping.

Decision Tree for Defining the Processing Mode

The figure below illustrates how the system defines one of the described processing modes based on the system properties described above:

Other Processing Modes

The DTP provides further processing modes for special applications and access methods:

Serial in the dialog process (for debugging)

With this processing mode you execute the data transfer process in debugging mode. The request is processed synchronously in a dialog process and the update of the data is simulated.

No data transfer; delta status in source: fetched

With this processing mode you execute a delta without transferring data. This is analogous to simulating the delta initialization with the InfoPackage. In this case you execute the DTP directly in the dialog.

Processing mode for real-time data packages

With this processing mode, you execute data transfer processes for real-time data acquisition.

Processing mode for direct access

With this processing mode, you execute data transfer processes for direct access.

Parallel extraction and processing except delta init.

Using this processing mode, you can achieve extensive parallelization for delta DTPs that extract data from a standard DataStore object who delta initialization is not read from the change log.

When the request is created for this kind of DTP, the processing mode is set dynamically as follows: During delta initialization, the data is extracted in serial form (from the active table with or without archive), and the data packages are processed in parallel processes. The delta requests are then extracted and processed in parallel.

Parallel extraction and processing (flexible preparation)

With this processing type, data transfer processes are executed that directly transfer data without a PSA from a DataSource (in the operational data provisioning (ODP) source system) to an InfoProvider. The processing mode is used if error handling is activated and the transformation does not require any semantic grouping.

The difference between this processing mode and V1 is that this processing mode has no explicit preparation phase in the program flow (see the Execute tab). During the preparation phase with ODP source systems, the system cannot find out how many data record and packages are delivered by the source. The main process does not ask whether a data package exists until after a parallel process has been spilt off.

SAP HANA Execution

If certain prerequisites are met, you can use this processing mode for a data transfer process that can be transformed in SAP HANA.

For more information, see Processing the Data Transfer Process in SAP HANA