Ingest MySQL Table

MySQL Consumer is based on datahub-flowagent and uses Data Services as an execution engine.

Data Services provides two methods of reading data from MySQL sources. You can either use MySQL Table Consumer to read the table, or use MySQL SQL Consumer to read the result set of the native SQL.

Table Consumer takes in schema.tablename as input, and it will read the table from the source piece-wise (N rows per call) and load to the next operator in line.

Prerequisites

Configuration Using Existing Client

MySQL operators require MySQL ODBC Connector for Linux x86-64.

Download MySQL ODBC Connector here. Select Linux - Generic and the operating system Linux - Generic (glibc 2.12) (x86, 64-bit) as the OS version, and then download the tar.gz.
Open Data Hub Launchpad > System Management > Files tab and create a folder in your Tenant Workspace named "flowagent".
Right-click the "flowagent" directory, select Import File and import the MySQL ODBC Connector tar.gz file.

Inside the newly created folder, create a new file named "tp-clients-env.sh" and put the following environment variables in there:

export MYSQL_ODBC_CONNECTOR=/vrep/flowagent/mysql-connector-odbc-8.0.13-linux-glibc2.12-x86-64bit
export MYSQL_5_5_DRIVERMANAGER=$MYSQL_ODBC_CONNECTOR/lib/libmyodbc8w.so
export MYSQL_5_6_DRIVERMANAGER=$MYSQL_ODBC_CONNECTOR/lib/libmyodbc8w.so

IMPORTANT: If the Flowagent application is already started, you must recreate it in order for the changes to be applied.

Components

Triggering the graph:

constantgenerator1: Provide constant input to consumer, which triggers the ingestion.
Defining Oracle Connection:
mysqltableconsumer1: Provides consumer connection information.

MySQL default port is 3306.

If no port is provided in connection definition, the default value will be assigned.
Loader Definition:
flowagentfileproducer1: Data Services requires a loader, such as CSV Loader, to load to Vflow operators