Confluent Kafka Sources for Replication Flows
You can use Confluent Kafka as source for your replication flows.
Prerequisites
-
Make sure you have created a confluent connection in Connection Management and that it is available in your space. For more information, see Confluent Connections
-
You must ensure that you connect to a supported Confluent/Kafka version. See the SAP Note 3806340
. -
All replication objects must have load type Initial and Delta (other load types are not supported).
Additional Considerations
-
You can transfer Kafka messages from Confluent Platform and Confluent Cloud topics to all supported targets for replication flows. Only Kafka messages whose message body is associated with a fixed schema that is registered in the Confluent schema registry can be replicated.
-
The source container that you select for the replication is a schema registry context in Confluent, and the replication objects are Confluent topics. The default context of the Confluent schema registry is displayed as period (.).
-
After you confirm your selection of replication objects, the system tries to determine the relevant schema for each object. The schema is defined by pulling the first message from the topic, fetching its schema ID, and extracting the schema from the selected context. If the schema ID is not found in the context, the replication fails and a context with the corresponding schema must be selected. If no message exists in the topic, the system tries to find a subject following the topic name strategy in the context. If nothing is found, you can still use the object, but have to configure the schema manually as described below.
-
When getting the data from the source, the system flattens any existing nested structures, as this is necessary for the purposes of the replication. To see the nested structures, choose Configure Schema in the side panel and switch from Flat to Nested.
-
By default, the __message_id column is used as the primary key for the purposes of the replication. The __message_id is constructed based on the Kafka partition the message was consumed from and the message offset of the corresponding Kafka message. The __message_id for a specific record then takes the form <partition>-<offset>, for example, 2-1 for a Kafka message with offset 1 that is part of partition 2.
Configuring Source Schemas
-
Open the relevant replication flow in the replication flow editor.
-
Select the relevant replication object.
-
Click Configure Schema in the side panel.
The configure schema dialog opens in 2 sections:-
The Properties section, where you can select properties that will affect the schema:
Properties
Description
Subject Name
Select the schema you want to use from the list of available values.
Schema Version
Select the schema version to use from the list of available values.
Expand Array
In case the selected Kafka Schema contains arrays or maps, you can select at most one field for expansion.When an array or a map is expanded, you must select at least 2 primary keys: one at the header level and one at the item level.
Omit Non-Expanded Arrays/Maps
When this option is enabled, all array/map fields that are not selected for expansion will be removed from the schema.
Include Technical Key
When the property Include Technical Key is selected, it ensures that the __message_id column is included as part of the selected table and treated as the primary key. If you deselect it, you hide the current primary key, and a new one must be selected.
Overwrite Source Settings at Object Level
By default, any settings that you have made at the replication object level are kept intact if you make a different setting at the replication flow level. To change this, enable this option.
-
The Schema section, where you will update the schema itself. It can be displayed in two different modes: Flat or Nested:
Properties Description Source columns (Technical Name)
Displays the technical name of the source column.
Source Data Type
Displays the column data type.
Opcode
You can define a source column to be interpreted as an operation code column. Only one column of data type String can be selected. Once Opcode is selected, you must define the opcode mapping.
Opcode Mapping
Define how the SAP opcodes column must be mapped with the values contained in your dataset for Insert, Update or Delete.
Key Column
One primary key must be defined. By default the message_id column is selected. However, you can choose a different primary key.
-
-
Click Apply Changes.
-
Save and redeploy
Source Settings and Additional Properties
The following properties are relevant for Confluent Kafka (in addition to the general properties). You can review and change these properties for the replication flow itself (by choosing ) or for individual replication objects (by selecting an object so that its properties get displayed in the side panel).
|
Property |
Description |
|---|---|
|
Consume Other Schema Versions |
Enable this option if you want the replication to consume other schema versions of the subject. By default, this property is deactivated, which means that the replication fails with an error if the system encounters a message whose version differs from the one it has determined for the topic. If you activate this property, the system automatically tries to consume other schema versions of the subject and match them with the schema that it determined earlier. |
|
Ignore Schema Mismatch |
[only visible if Consume Other Schema Versions is enabled] Enable this option if you want the replication to continue even when the schemas do not match. By default, this property is deactivated, which means that the replication fails with an error if it encounters a message with a different schema. If you activate this property, the system ignores messages whose schema differs from the previously determined one (for example if it has a different serialization format). Note that the replication always fails (irrespective of the setting you make here) if the different schemas belong to different subjects, have different key columns, or include a column whose data type is different for the different schemas. |
|
Fail on Data Truncation |
Enable this option if you want the replication to fail when the source data is too long. By default, the replication fails if a value exceeds either the field length for STRING (5000) or BINARY(5000) or the specified precision. If you deactivate this property, source values that are too long are automatically truncated. |
|
Starting Point |
Select from where to start the reading operation. By default, the topics are read starting from the oldest available partition offsets (Read from earliest offset). Alternatively, you can select Read from latest offset. If you do so, the system reads only messages newer than the ones present when the replication starts the delta mode phase. |
For more information, see the blog Confluent as Replication Source
Data Types
|
SAP Datasphere |
AVRO Data Type |
JSON Data Type |
|---|---|---|
|
binary |
bytes |
String (base64-encoded) |
|
boolean |
boolean |
Boolean |
|
date |
int (date) |
String (YYYY-MM-DD) |
|
decimal(p,s) |
bytes (DECIMAL p,s) |
Number |
|
decfloat16 |
bytes (DECIMAL 28,6) |
Number |
|
decfloat34 |
bytes (DECIMAL 38,6) |
Number |
|
float32 |
float |
Number |
|
float64 |
double |
Number |
|
int8 |
int |
Number |
|
int16 |
int |
Number |
|
int32 |
int |
Number |
|
int64 |
long |
Number |
|
string |
string |
String |
|
time |
long (timestamp-micros) |
String (HH:MM:SS.NNNNNNNNNN) |
|
timestamp |
long (timestamp-micros) |
String (YYYY-MM-DD HH:MM:SS.NNNNNNNNNN) |
|
uint8 |
int |
Number |
|
uint64 |
bytes (DECIMAL 20,0) |
Number |