Confluent Kafka Sources for Replication Flows

You can use Confluent Kafka as source for your replication flows.

Prerequisites

  • Make sure you have created a confluent connection in Connection Management and that it is available in your space. For more information, see Confluent Connections

  • You must ensure that you connect to a supported Confluent/Kafka version. See the SAP Note 3806340 Information published on SAP site.

  • All replication objects must have load type Initial and Delta (other load types are not supported).

Additional Considerations

If you use Confluent Kafka as the source for your replication flow, you need to consider the following additional specifics and conditions:
  • You can transfer Kafka messages from Confluent Platform and Confluent Cloud topics to all supported targets for replication flows. Only Kafka messages whose message body is associated with a fixed schema that is registered in the Confluent schema registry can be replicated.

  • The source container that you select for the replication is a schema registry context in Confluent, and the replication objects are Confluent topics. The default context of the Confluent schema registry is displayed as period (.).

  • After you confirm your selection of replication objects, the system tries to determine the relevant schema for each object. The schema is defined by pulling the first message from the topic, fetching its schema ID, and extracting the schema from the selected context. If the schema ID is not found in the context, the replication fails and a context with the corresponding schema must be selected. If no message exists in the topic, the system tries to find a subject following the topic name strategy in the context. If nothing is found, you can still use the object, but have to configure the schema manually as described below.

  • When getting the data from the source, the system flattens any existing nested structures, as this is necessary for the purposes of the replication. To see the nested structures, choose Configure Schema in the side panel and switch from Flat to Nested.

  • By default, the __message_id column is used as the primary key for the purposes of the replication. The __message_id is constructed based on the Kafka partition the message was consumed from and the message offset of the corresponding Kafka message. The __message_id for a specific record then takes the form <partition>-<offset>, for example, 2-1 for a Kafka message with offset 1 that is part of partition 2.

Configuring Source Schemas

When you select a replication object, its properties, including schema-related information, are displayed in the side panel. To change the schema-related properties,
  1. Open the relevant replication flow in the replication flow editor.

  2. Select the relevant replication object.

  3. Click Configure Schema in the side panel.

    The configure schema dialog opens in 2 sections:
    • The Properties section, where you can select properties that will affect the schema:

      Properties

      Description

      Subject Name

      Select the schema you want to use from the list of available values.

      Schema Version

      Select the schema version to use from the list of available values.

      Expand Array

      In case the selected Kafka Schema contains arrays or maps, you can select at most one field for expansion.

      When an array or a map is expanded, you must select at least 2 primary keys: one at the header level and one at the item level.

      Omit Non-Expanded Arrays/Maps

      When this option is enabled, all array/map fields that are not selected for expansion will be removed from the schema.

      Include Technical Key

      When the property Include Technical Key is selected, it ensures that the __message_id column is included as part of the selected table and treated as the primary key. If you deselect it, you hide the current primary key, and a new one must be selected.

      Overwrite Source Settings at Object Level

      By default, any settings that you have made at the replication object level are kept intact if you make a different setting at the replication flow level. To change this, enable this option.

    • The Schema section, where you will update the schema itself. It can be displayed in two different modes: Flat or Nested:

      Properties Description

      Source columns (Technical Name)

      Displays the technical name of the source column.

      Source Data Type

      Displays the column data type.

      Opcode

      You can define a source column to be interpreted as an operation code column. Only one column of data type String can be selected. Once Opcode is selected, you must define the opcode mapping.

      Opcode Mapping

      Define how the SAP opcodes column must be mapped with the values contained in your dataset for Insert, Update or Delete.

      Key Column

      One primary key must be defined. By default the message_id column is selected. However, you can choose a different primary key.

  4. Click Apply Changes.

  5. Save and redeploy

Source Settings and Additional Properties

The following properties are relevant for Confluent Kafka (in addition to the general properties). You can review and change these properties for the replication flow itself (by choosing (Browse source settings)) or for individual replication objects (by selecting an object so that its properties get displayed in the side panel).

Property

Description

Consume Other Schema Versions

Enable this option if you want the replication to consume other schema versions of the subject.

By default, this property is deactivated, which means that the replication fails with an error if the system encounters a message whose version differs from the one it has determined for the topic. If you activate this property, the system automatically tries to consume other schema versions of the subject and match them with the schema that it determined earlier.

Ignore Schema Mismatch

[only visible if Consume Other Schema Versions is enabled] Enable this option if you want the replication to continue even when the schemas do not match.

By default, this property is deactivated, which means that the replication fails with an error if it encounters a message with a different schema. If you activate this property, the system ignores messages whose schema differs from the previously determined one (for example if it has a different serialization format). Note that the replication always fails (irrespective of the setting you make here) if the different schemas belong to different subjects, have different key columns, or include a column whose data type is different for the different schemas.

Fail on Data Truncation

Enable this option if you want the replication to fail when the source data is too long. By default, the replication fails if a value exceeds either the field length for STRING (5000) or BINARY(5000) or the specified precision. If you deactivate this property, source values that are too long are automatically truncated.

Starting Point

Select from where to start the reading operation.

By default, the topics are read starting from the oldest available partition offsets (Read from earliest offset). Alternatively, you can select Read from latest offset. If you do so, the system reads only messages newer than the ones present when the replication starts the delta mode phase.

For more information, see the blog Confluent as Replication SourceInformation published on SAP site

Data Types

The following table shows how source data types are mapped to internal data types. The source data type depends on the serialization format of the topic.

SAP Datasphere

AVRO Data Type

JSON Data Type

binary

bytes

String (base64-encoded)

boolean

boolean

Boolean

date

int (date)

String (YYYY-MM-DD)

decimal(p,s)

bytes (DECIMAL p,s)

Number

decfloat16

bytes (DECIMAL 28,6)

Number

decfloat34

bytes (DECIMAL 38,6)

Number

float32

float

Number

float64

double

Number

int8

int

Number

int16

int

Number

int32

int

Number

int64

long

Number

string

string

String

time

long (timestamp-micros)

String (HH:MM:SS.NNNNNNNNNN)

timestamp

long (timestamp-micros)

String (YYYY-MM-DD HH:MM:SS.NNNNNNNNNN)

uint8

int

Number

uint64

bytes (DECIMAL 20,0)

Number