Cloud Storage Provider Sources for Replication Flows

If you use a cloud storage provider as the source for your replication flow, you need to consider additional specifics and conditions.

Prerequisites

  • All objects in your replication flow have load type Initial Only.
  • All objects must have one of the following formats:
    • CSV files, with one of the following delimiters:

      • Comma (,)

      • Semicolon (;)

      • Colon (:)

      • Vertical slash (|)

      • Horizontal tabulator (also known as tab or \t)

    • Parquet files

Additional Properties

The following properties are relevant for cloud storage providers as sources for a replication flow. You can review and change these properties for the replication flow itself (by choosing (Browse source settings)) or for individual replication objects (by selecting an object so that its properties get displayed in the side panel).

Property

Description

Fail on Data Truncation

Enable this option if you want the replication to fail if the source data is too long. By default, the replication fails if it encounters a string value greater than 5000, a binary value greater than 3000, or a decimal value with precision between 1 and 76 (1 <= precision <= 76) and scale between 0 and precision (0 <= scale <= precision). If you deactivate this property, source values that are too long are automatically truncated.

Fail on Incompatible Data

Enable this option if you want the replication to fail when it encounters values that are incompatible with the data type of the relevant target column. By default, the replication fails if it encounters incompatible data. If you deactivate this option, the system replaces incompatible values with NULL values.

Define Max. Number of Partitions

Enable this option if you want to define the maximum number of partitions manually. By default, the system uses 10 partitions. If you activate this option, an additional input field is displayed where you can enter the maximum number of partitions.

Max. Number of Partitions

Enter the maximum number of partitions to be used during the replication. By default, the system uses a maximum of 10 partitions for a replication object. However, depending on the specifics of your replication flow it may make sense to change this value (for example if you use filtering so that only a fraction of the data is actually relevant for replication). You can enter any integer number between 1 and 2147483647.

Include Subfolders

Enable this option if you want data that is stored in subfolders to be included in the replication. By default, the data in all subfolders of the selected CSV files is considered for replication, and there is no maximum depth for nesting. If you disable this option, data in subfolders is ignored during the replication.

Global Pattern

Enter a global pattern if you want to only replicate files whose names conform with the pattern. See below for detailed examples.

Defining the Schema (and Related Properties)

In the step for selecting the source objects for your replication flow, you get a list of all files in the relevant folder. You then select the file whose metadata (particularly the schema) you want to use for the target.

You can edit the schema later if necessary. To do so, select the object whose schema you want to change, then choose Edit Schema (in the properties panel). A list of the source columns with their respective data type and other relevant properties (such as precision and scale) is displayed. Change these values as required, then choose Update Schema.
In addition, you can change the following other properties here:
  • File Delimiter: The existing delimiter is automatically entered as the default value. You can change it to any of the allowed values as listed above.

  • Encoding: The existing encoding value is automatically entered as the default default value. Alternatively (if the system cannot determine the existing value), UTF-8 is used as the default. You can change it to UTF-16 or ISO-8859-1.

  • File Header: By default, the system assumes that the first row in each CSV file is a header. If this is not the case, deactivate this property. The system then automatically names the columns as C(index), starting with C0.

Replicate Source Objects Without Primary Keys

If a source object does not have a primary key, you can still use it in a replication flow by configuring the primary key columns manually. The target table requires a primary key, so you must define which columns serve as the key before you deploy the replication flow. To configure the primary key columns:
  1. Select the relevant source object.

  2. In the properties panel, go to the Source Schema Settings section and choose Configure Schema.

  3. Mark the relevant columns as key columns.

The columns you select as the primary key must contain unique, non-null values. Incorrect settings result in data inconsistency.

Examples for Global Patterns

You have the following files:

  • /datasetFolder/example.csv

  • /datasetFolder/test1.csv

  • /datasetFolder/test2.csv

  • /datasetFolder/test3.json

  • /datasetFolder/subfolder1/test4.csv

  • /datasetFolder/subfolder1/subfolder2/test5.csv

  • /datasetFolderB/example.parquet

  • /datasetFolderB/test1.parquet

  • /datasetFolderB/test2.parquet

  • /datasetFolderB/test3.parquet

  • /datasetFolderB/subfolder1/test4.parquet

  • /datasetFolderB/subfolder1/subfolder2/test5.parquet

Example 1:

You define a global pattern as “test*.csv”. The result is that only the matching files from the dataset folder are picked up (regardless of whether the configuration property Include Subfolders is enabled or disabled):

  • /datasetFolder/test1.csv

  • /datasetFolder/test2.csv

Example 2:

You define the global pattern as “**/test*.csv” and enable Include Subfolders. Then the result looks like this:

  • /datasetFolder/test1.csv

  • /datasetFolder/test2.csv

  • /datasetFolder/subfolder1/test4.csv

  • /datasetFolder/subfolder1/subfolder2/test5.cs

If Include Subfolders is disabled, the result is as follows, even if the global pattern allows going to the subfolders:

  • /datasetFolder/test1.csv

  • /datasetFolder/test2.csv

Example 3:

You define a global pattern as “test*.parquet”. The result is that only the matching files from the dataset folder are picked up (regardless of whether the configuration property Include Subfolders is enabled or disabled):

  • /datasetFolderB/test1.parquet

  • /datasetFolderB/test2.parquet

If you enable Include Subfolders, then the result looks like this:

  • /datasetFolderB/test1.parquet

  • /datasetFolderB/test2.parquet

  • /datasetFolderB/subfolder1/test4.parquet

  • /datasetFolderB/subfolder1/subfolder2/test5.parquet

Example 4:

You define 2 global patterns as “test*.csv” and “test*.parquet”. The result is that only the matching files from the dataset folder are picked up (regardless of whether the configuration property Include Subfolders is enabled or disabled):

  • /datasetFolder/test1.csv

  • /datasetFolder/test2.csv

  • /datasetFolderB/test1.parquet

  • /datasetFolderB/test2.parquet

If you enable Include Subfolders. Then the result looks like this:

  • /datasetFolder/test1.csv

  • /datasetFolder/test2.csv

  • /datasetFolder/subfolder1/test4.csv

  • /datasetFolder/subfolder1/subfolder2/test5.csv

  • /datasetFolderB/test1.parquet

  • /datasetFolderB/test2.parquet

  • /datasetFolderB/subfolder1/test4.parquet

  • /datasetFolderB/subfolder1/subfolder2/test5.parquet

Data Types

The following table shows how source data types are mapped to internal data types for cloud storage provider sources. The source data type depends on the file format: Parquet or CSV.

SAP Datasphere

Parquet Data Type

CSV Representation

binary

BYTE_ARRAY

Base64-encoded

boolean

BOOLEAN

true or false

date

INT32 (DATE)

YYYY-MM-DD

decimal(p,s)

DECIMAL(p,s)

As is

decfloat16

DECIMAL(28,6)

As is

decfloat34

DECIMAL(38,6)

As is

float32

FLOAT

Decimal or scientific notation

float64

DOUBLE

Decimal or scientific notation

int8

INT32

Integer (base 10)

int16

INT32

Integer (base 10)

int32

INT32

Integer (base 10)

int64

INT64

Integer (base 10)

string

BYTE_ARRAY (STRING)

As is

time

INT64 (TIME microseconds)

HH:MM:SS.NNNNNNNNNN

timestamp

INT64 (TIMESTAMP microseconds)

YYYY-MM-DD HH:MM:SS.NNNNNNNNNN

uint8

INT32

Integer (base 10)

uint64

INT64

Integer (base 10)