Add a Source to a Data Flow

Add a source to read data from. You can add multiple sources and combine them together using join or union operators.

Procedure

  1. If the Source Browser panel is not visible on the left of the screen, click Source Browser in the toolbar to show it.
  2. Browse or search for the object you want to add on either of the tabs.
    • The Repository tab lists all the tables, views, and intelligent lookups that are available in the space (including objects shared to the space)..

      For more information, see Add Objects from the Repository.
    • The Sources tab lists all the connections and other data sources that have been integrated to the space from which you can import tables. However it shows only limited records. If you can't see the sources you are looking for, use Import from Connection to perform search. You can:

  1. Select the object of your choice, and then drag and drop it onto the diagram.
  2. Click the source node to display its properties in the side panel, and complete the properties in the General section:

    Property

    Description

    Label

    Enter a label to display in the source symbol.

    Business Name / Technical Name / Type / Connection

    [read-only] Provide information to identify the source table.

    Package

    Select the package to which the object belongs.

    Packages are used to group related objects in order to facilitate their transport between tenants. For more information, see Creating Packages to Export.

    Type

    Type of object. For example: a local table.

    Status

    [read-only] Displays the deployment and error status of the object.

    For more information, see Saving and Deploying Objects.

    Use As

    [read-only] Specifies whether the table is used as a Source or Target in the data flow.

  3. In the Columns section, review the columns of the source.
    • To view the data type of a column, hover over it and click

    • If the source is a CSV, JSON, or Excel file, you can change the data type of a column and its corresponding properties by hovering over the column and clicking (Menu)Start of the navigation path Edit ColumnEnd of the navigation path.

    • For a remote source, you can control the data being read.

      • To add/delete columns from source, click / (Delete) respectively.

      • To add new/removed columns from the source based on latest metadata, click (Refresh to fetch column changes from source)

  4. [optional] For CDI, OData, Open Connectors and ABAP CDS sources complete the Source Filters section to define filter conditions for the consumption of their data:
    When defining filter conditions for columns,the following rules are applied:
    • An "OR" is applied between the conditions defined on the same column.

    • An "AND" is applied between the conditions defined on different columns.

    For SAP SuccessFactors, OData, and Cloud Data Integration (CDI) sources, you can enable delta loading. That is, move over the changed data records instead of doing a full load every time.

    To do this, in the Source Filters dialog, you must manually enter the condition $MAX_VALUE_<column_name> in the Value field.

    <column_name> is a column in the target table where data is being replicated, normally of type date or timestamp.

  5. [optional] For source tables from OData remote connections, you can add or edit the parameters of a selected source table in the OData Parameters section. OData parameters allow you to define additional information or constraints to control how the data is returned from the OData connection. Click (Define Parameters) to open the OData Parameters dialog. You can:
    • Manually enter a custom parameter according to URI conventions (see URI ConventionsInformation published on non-SAP site).

    • Click the Parameters dropdown button to see a list of suggested parameters.

    Add a value and click OK to validate your changes or Reset to erase all parameters.

    In the OData Parameters section, enable the Apply Parameters on Preview toggle if you want to see how the changes in parameter impact the table preview.

  6. [optional] For source tables from OData remote connections, you can edit the depth properties of a selected source table in the OData Properties section. Click (Edit Data Properties) to open the OData Properties dialog. You can set the depth of the source table to either 1 or 2.

    The depth of an OData object refers to the level of related entities that are included in the response when querying the OData service. The depth is by default set to 1 so that only the properties of the requested entity are returned. You can change the depth to 2 to include a second level. Depth is useful when you want to optimize performance by controlling the amount of data returned in a single request.

  7. [optional] For sources that are files, complete the appropriate additional section:
    • For .csv files, click Modify in the CSV Properties section to configure various csv properties for your .csv file (column separator, text separator, character set, escape character, etc.).

    • For .json and .jsonl files, click Modify in the JSON Properties section to configure the flattening levels of the JSON file.

    • For .xls and .xlsx files, click Modify in the Excel Properties to choose the Excel worksheet to be used and to set a header row.

  8. [optional] In the Advanced section, configure the following properties:

    Property

    Description

    Control Fetch Size

    To enforce a specific value for Batch Size, toggle this switch.

    Fetch Size (Number of Rows)

    Specifies the amount of data being read. It indicates the number of rows that will be committed to target on each batch.

    Batch Query

    Fetches the data batch by batch according to the fetch size defined above. Use this option for very large odata source files.

    Fail Run on String Truncation

    Fails the data flow run if string truncation is detected while fetching the source columns. This property is available only for CSV, JSON, and Excel files.

    Enable ODBC Tracing

    : [SAP HANA connection only] Enable this option to create a new log file in the vflow graph pod with HANA ODBC debug logs.

    Access Method

    Select the open connector API to use to access the data:
    • GET: The Open Connectors data query API is called directly once per page to retrieve data. It's the recommended default method, except in cases of large amounts of data.

    • BULK: An asynchronous query job is created on Open Connectors to start streaming the data. It's better to use this method to get better performance in cases of large amounts of data.

  9. To finalize the source, create a flow from it to an operator or to the target table, as appropriate.