Modeling Guide for SAP Data Hub

Ingest DB2 Table

DB2 Consumer is based on datahub-flowagent and uses Data Services as an execution engine.

Data Services provides two methods of reading data from DB2 sources. You can either use DB2 Table Consumer to read the table or use DB2 SQL Consumer to read the result set of the native SQL.

DB2 Table Consumer

Table Consumer takes in the schema.tablename as input and it will read the table from the source piece-wise (100 rows per call), and load to the next operator in line.

Prerequisites

Configuration using DB2 ODBC CLI Driver

DB2 operators require DB2 ODBC CLI Driver for Linux.

  • Download DB2 ODBC CLI Driver hereInformation published on non-SAP site. Select IBM Data Server Driver for ODBC and CLI (CLI Driver), log in to your IBM account, and then select the Linux platform (only Linux).

  • Open Data Hub Launchpad > System Management > Files tab and create a folder in your Tenant Workspace named "flowagent".

  • Right-click the "flowagent" directory, select Import File and import the DB2 ODBC CLI Driver tar.gz file.

  • Inside the newly created folder, create a new file named "tp-clients-env.sh" and put the following environment variables in there:
    export DB2_CLI_DRIVER=/vrep/flowagent/clidriver/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$DB2_CLI_DRIVER
  • IMPORTANT: If the Flowagent application is already started, you must recreate it in order for the changes to be applied.

Components

  • Defining DB2 Connection:

    db2sqlconsumer1: provides DB2 consumer connection information.

    DB2 default port is 50000.

    If no port is provided in connection definition, the default value will be assigned.

  • Loader definition:

    flowagentfileproducer1: Data Services requires a loader, such as CSV Loader, to load to Vflow operators.