Modeling Guide for SAP Data Hub

Ingest MySQL SQL

MySQL Consumer is based on datahub-flowagent and uses Data Services as an execution engine.

Data Services provides two methods of reading data from MySQL sources. You can either use MySQL Table Consumer to read the table, or use MySQL SQL Consumer to read the result set of the native SQL.

SQL Consumer takes in native SQL statement that you would run on MySQL as input, and it will read the response result set from the source piece-wise (N rows per call) and load to the next operator in line.

Prerequisites

See MySQL ODBC Driver configuration on the Ingest MySQL Table graph (com.sap.dh.ingestion.mysql.tableconsumer).

Components

  • Triggering the graph:

    constantgenerator1: Provide constant input to consumer, which triggers the ingestion.

  • Defining Connection:

    mysqlsqlconsumer1: Provides consumer connection information.

    MySQL default port is 3306.

    If no port is provided in connection definition, the default value will be assigned.

  • Loader Definition:

    flowagentfileproducer1: Data Services requires a loader, such as CSV Loader, to load to Vflow operators

Partition

Please see partition configuration on the Oracle SQL Consumer graph (com.sap.dh.ingestion.oracle.sqlconsumer).