MySQL Sources for Replication Flows

You can use MySQL connections as sources in replication flows to replicate data into supported targets. This feature uses the Connectivity Framework and supports the Initial Only load type.

Prerequisites

  • Make sure you have created a MySQL connection in Connection Management and that it is available in your space. For more information, see MySQL Connections

  • Only tables that contain a primary key are supported as replication sources.

  • Only the Initial Only load type is supported. Initial and Delta and Delta Only load types are not supported.

Restrictions

  • Source settings are not available for this source type.

  • Custom parameters are not supported.

  • Connections through load balancers are not supported.

Partitioning Timeouts for Large MySQL Tables

If a partitioning timeout occurs, try the following in order:

  • Stale table statistics: MySQL statistics in information_schema can take time to update. Stale statistics lead to incorrect group calculations and a partitioning timeout. Run the following statement, then rerun the replication ANALYZE TABLE <tableIdentifier>;flow:

  • Persistent timeout – InnoDB tuning: If the timeout persists, review the following MySQL parameters to ensure the index fits in memory and I/O is tuned for sequential large-table reads:

    Parameter

    Recommended Value

    innodb_buffer_pool_size

    70–80% of RAM (dedicated server); at minimum, twice the index size

    innodb_buffer_pool_instances

    1 per GB of buffer pool; maximum 8

    innodb_read_io_threads

    8 for SSD; 16 or more for NVMe (restart required)

    innodb_read_ahead_threshold

    0 for sequential replication scans

    read_rnd_buffer_size

    32 MB for large table reads

    max_connections

    Set conservatively; each idle connection uses approximately 1 MB

  • Continued DEADLINE_EXCEEDED errors – covering index: On very large tables, create the SAP_MYSQL_REPLICATION_PK index to allow the connector to scan only the primary key columns instead of the full clustered index. The index must cover exactly the primary key columns and nothing else:-- Replace pk1, pk2, ... with the actual primary key column names in declaration order. CREATE INDEX SAP_MYSQL_REPLICATION_PK ON `schema`.`table` (`pk1`, `pk2`, ...);

    • Index creation runs online but takes time proportional to table size. Monitor progress with:SELECT STAGE, WORK_COMPLETED, WORK_ESTIMATED FROM performance_schema.events_stages_current;

    • After creation, verify the index is used:EXPLAIN SELECT `pk1`, ..., `pkN` FROM `schema`.`table` ORDER BY `pk1`, ..., `pkN` LIMIT 1 OFFSET 1000000; -- Expected: key = SAP_MYSQL_REPLICATION_PK, Extra = Using index

    • Verify the index size fits within your innodb_buffer_pool_size:SELECT INDEX_NAME, ROUND(stat_value * @@innodb_page_size / 1024 / 1024 / 1024, 2) AS size_gb FROM mysql.innodb_index_stats WHERE database_name = '<schema>' AND table_name = '<table>' AND stat_name = 'size';

Replicating Source Objects Without Primary Keys

Replicating source objects without a primary key is not supported. Only source objects that have a primary key defined can be used as source objects in a replication flow.

Data Types

The following table shows how MySQL source data types are mapped to internal data types. Data types that are not supported cannot be replicated.

MySQL Data Type

SAP Datasphere

BIGINT

int64

BIGINT UNSIGNED

uint64

BINARY(n), VARBINARY(n), 1≤n≤5000

binary(n)

BINARY(n), VARBINARY(n), n > 5000

Not Supported

BIT(1)

bool

BIT(n > 1)

binary(n)

BLOB, MEDIUMBLOB, LONGBLOB

Not Supported

BOOL, BOOLEAN

int8

CHAR(n), NCHAR(n), 1≤n≤5000

string(n)

CHAR(n), NCHAR(n), n > 5000

Not Supported

DATE, YEAR

date

DECIMAL (unknown precision/scale)

decfloat34

DECIMAL(p,s)

decimal(p,s)

DOUBLE, REAL

float64

ENUM, SET

Not Supported

FLOAT

float32

GEOMETRY, GEOMETRYCOLLECTION, LINESTRING, MULTILINESTRING, MULTIPOINT, MULTIPOLYGON, POINT, POLYGON

Not Supported

INT, MEDIUMINT

int32

INT UNSIGNED, MEDIUMINT UNSIGNED

uint64

JSON

Not Supported

SMALLINT

int16

SMALLINT UNSIGNED

uint64

TEXT, MEDIUMTEXT, LONGTEXT, 1≤n≤5000

string(n)

TEXT, MEDIUMTEXT, LONGTEXT, n > 5000

Not Supported

TIME

time

TIMESTAMP(n), DATETIME(n)

timestamp

TINYBLOB

binary(n)

TINYINT

int8

TINYINT UNSIGNED

uint8

TINYTEXT

string(n)

VARCHAR(n), NVARCHAR(n), 1≤n≤5000

string(n)

VARCHAR(n), NVARCHAR(n), n > 5000

Not Supported