Creating a Data Flow

Create a data flow to move and transform data in an intuitive graphical interface. You can drag and drop sources from the Source Browser, join them as appropriate, add other operators to remove or create columns, aggregate data, and do Python scripting, before writing the data to the target table.

Prerequisites

To create flows, you must have a scoped role that grants you access to a space with the following privileges:
  • Data Warehouse General (-R------) - To access SAP Datasphere.

  • Data Warehouse Connection (-R------) - To access remote objects.

  • Data Warehouse Data Builder (CRUD----) - To create, edit, and delete flows.

  • Space Files (CRUD----) - To create, read, update, and delete objects in your spaces.

To run and schedule flows, you must, in addition, have the following privileges:
  • Data Warehouse Data Integration (-R------) - To view data integration task logs in the Data Integration Monitor app.

  • Data Warehouse Data Integration (--U-----) - To manually run data integration tasks.

  • Data Warehouse Data Integration (----E---) - To schedule data integration tasks.

The DW Modeler role template, for example, grants the privileges to create and manage flows, and the DW Integrator role template grants the privileges to run them. For more information, see Privileges and Permissions and Standard Roles Delivered with SAP Datasphere.

Context

In addition to working with flows in the editor, you can also:

Procedure

  1. In the side navigation area, select (Data Builder), select a space if necessary, and select New Data Flow to open the editor.
  2. Drag one or more source objects from the Source Browser and drop it into the diagram (see Add a Source to a Data Flow).
  3. Transform your data using one or more operators:
  4. Select an object in the canvas to display its properties in the side panel and to reveal its contextual toolbar, which contains some or all of the following tools:

    Tool

    Description

    (Create Flow)

    Click to create a flow to another operator in the data flow. Drag and drop the yellow dotted line to the required operator to which you want to connect.

    (Preview Data)

    Click to preview data of the selected operator. The Data Preview section is displayed.

    (Add Table)

    Create a new target table.

    (Impact and Lineage Analysis)

    Open the Impact and Lineage Analysis diagram. This diagram enables you to understand the lineage and impacts of the selected object.

    (see Impact and Lineage Analysis.)

    (Open in New Tab)

    Open the selected entity in its own editor in a new tab.

    (Remove)

    Click to delete the selected operator.

  5. Add or create a target table that the data flow will write its data to.

    For more information, see Add or Create a Target Table in a Data Flow.

  6. Click on the canva and review your data flow properties in the right panel:
    • Under General:
      Property Description

      Business Name

      Enter a descriptive name to help users identify the object. This name can be changed at any time.

      Technical Name

      Displays the name used in scripts and code, synchronized by default with the Business Name.

      To override the default technical name, enter a new one in the field. Technical names can contain only alphanumeric characters and underscores.

      Package

      Select the package to which the object belongs.

      Packages are used to group related objects in order to facilitate their transport between tenants. For more information, see Creating Packages to Export.

      Status

      [read-only] Displays the deployment and error status of the object.

      For more information, see Saving and Deploying Objects.

    • Under Run Status:

      Displays the status of the flow run:

      • Running: The flow is currently running.

      • Completed: The flow is completed successfully.

      • Failed: Something goes wrong during the flow run and it could not be completed. Go to the details screen of your flow run and check the logs to identify to issue.

    • Under Input Parameters:Create a new input parameter or modify an existing one. For more information, see Create an Input Parameter in a Data Flow

    • Under Advanced Properties

      • Dynamic Memory Allocation You can allocate memory usage manually. Set the Expected Data Volume to Small, Medium, or Large.

      • Automatic restart on run failure: Set this option to restart the data flow automatically if there are any failures or system upgrades, for example.

  7. Click (Save) to save your data flow:
    • Save to save the data flow.

    • Save As to create a local a copy of the data flow. The data flow must have been previously saved at least once. The Save dialog opens. Enter new business and technical names and click Save.

  8. Click (Deploy) to deploy the data flow:
    • Newly created data flows are deployed for the first time.

    • Data flows that have changes to deploy are redeployed.

    With deployment, you will be able to save draft version of your data flow without affecting the execution.

  9. The tools in the editor toolbar help you work with your object throughout its lifecycle:

    Tool

    Description

    (Save)

    Save your changes to the design-time repository. You can use Save As to create a copy of the object.

    See Saving and Deploying Objects.

    (Deploy)

    Deploy your changes to make them available in the run-time environment.

    See Saving and Deploying Objects.

    (Run)

    Run the object to obtain or update its output results.

    You must deploy the object before you can run it.

    (Source Browser)

    Show the Source Browser panel to drag sources into the editor.

    See Using the Source Browser.

    (Preview Data)

    Open the Data Preview panel to preview data in the selected diagram node.

    See Viewing Object Data.

    (Undo) / (Redo)

    Revert the last change to the object or redo a change you have previously undone.

    (Export)

    Export the object to a CSN/JSON file.

    See Exporting Objects to a CSN/JSON File.

    (Impact and Lineage Analysis)

    Open the Impact and Lineage Analysis graph for the object.

    See Impact and Lineage Analysis.