Creating a Task Chain

Group multiple tasks into a task chain and run them manually once, or periodically, through a schedule.

Prerequisites

To create task chains, you must have a scoped role that grants you access to a space with the following privileges:
  • Data Warehouse General (-R------) - To access SAP Datasphere.

  • Data Warehouse Data Builder (CRUD----) - To create, edit, and delete task chains.

  • Space Files (CRUD----) - To create, read, update, and delete objects in your spaces.

To run and share task chains and configure email notifications, you must, in addition, have the following privileges:
  • Data Warehouse General (-R------) - To access SAP Datasphere.

  • Data Warehouse Data Integration (-R------) - To view data integration task logs in the Data Integration Monitor app.

  • Data Warehouse Data Integration (--U-----) - To manually run data integration tasks.

  • Data Warehouse Data Integration (----E---) - To schedule data integration tasks.

  • Data Warehouse Data Builder (------S-) - To share task chains to other spaces.

  • User (R-------) - To display and add notification recipients from a list of current tenant members, when setting up email notifications.

The DW Modeler role template, for example, grants the privileges to create and manage task chains, and the DW Integrator role template grants the privileges to run and share them. For more information, see Privileges and Permissions and Standard Roles Delivered with SAP Datasphere.
The following prerequisites also apply:
  • For SAP HANA Open SQL schema procedures to be available for users to include in a task chain, the schema’s owner must grant EXECUTE privileges to the space user for objects in the Open SQL schema. See Allow the Space to Access the Open SQL Schema.

  • Objects must have been already deployed, so that they can be added to the task chain. Task chains must also be deployed to allow selection of tenant users or specify email addresses for notification of task chain completion.

  • Persisting of views may include one parameter that uses the default value defined with the view. Views must not have data access controls assigned to them.

  • If a data flow that has input parameters is included in a task chain, task chain runs will use default parameter values defined for the data flow.

  • You can create a schedule for your replication flow and include it in a task chain. For replication flows with Initial and Delta and Delta Only load types, you must set Delta Load Run to At Scheduled Time.

Context

You can create task chains that include SAP Datasphere repository objects, that is, Remote Tables and Views, Local Tables, Intelligent Lookups, Data Flows, Replication Flows, and Transformation Flows. You can also include non-repository objects such as SAP HANA Open SQL schema procedures and SAP BW Bridge process chains. In addition, you can nest other existing, locally-created or shared task chains in other task chains, as well as share task chains you've created to other spaces.

In addition to the objects available from the Repository or Others tabs, you can also add two other additional objects to task chains that are only available from the task chain toolbar. The API Task object lets you configure and run API asks to access external systems. The Notification Task object lets you configure email notification for individual task chain tasks.

When creating a task chain, you can create linear task chains in which one task is run after another. A succeeding task is only run once the previous task in the series has finished successfully with a completed status. The running of tasks in the series may resume if the previous task has a failed status. The task port connecting one object task to the next object task can allow the task chain to continue if a task regardless of the failed. For more information about task ports, see Select Task Ports in a Task Chain. You can also create task chains in which individual tasks are run in parallel and successful continuation of the entire task chain run depends on whether ANY or ALL parallel tasks are completed successfully.

You can turn on the Ignore Error button in the right side panel to disregard the status of the single object task in the status evaluation of the entire task chain.

When creating or editing a task chain, you can also set up email notification for deployed task chains to notify selected users of task chain completion. After deploying a task chain, you can add tenant users or email addresses to notify individuals when task chain runs are completed.

You can monitor the status of task chain runs from the Data Integration Monitor. For more information, see Monitoring Task Chains.

In addition to working with task chains in the editor, you can also:

Procedure

  1. From the Data Builder, click New Task Chain.
  2. From the left-side panel, drag and drop a first object on to the task chain canvas from those available in the repository (from the Repository tab), or non-repository objects (selected from the Others tab).
    • From the Repository tab, you can see the remote tables, views, intelligent lookups, data flow, replication flow, transformation flow objects and task chains that meet prerequisites and are available to be added to the task chain. Task chains you've shared or have been shared with you from another space are denoted by the (Share) icon following the task chain's business name. For more information on sharing and running shared task chains, see Nest and Share Task Chains.

    • From the Others tab, you can see the non-repository Open SQL schema procedures and BW Bridge process chains you can add to a task chain. For more information on adding Open SQL schema procedures from the Others tab, see Run Open SQL Procedures in a Task Chain. For more information on adding BW Bridge process chains to a task chain, see Run SAP BW Bridge Process Chains in a Task Chain.

    • In addition to the objects available from the Repository or Others tabs, you can also add two other additional objects to task chains that are only available from the task chain toolbar. The API Task object lets you configure API Tasks to access external systems. (For more information, see Run API Tasks in a Task Chain.) The Notification Task object lets you configure email notification for individual task chain tasks. (For more information, see Configure Email Notification for Task Chains and Notification Tasks.)

  3. Drag a second object onto the first object in the task chain. As you drag the object over the top of the first object, a context menu displays options Add as New Task (the default), Replace Existing, or Add as Parallel to place the new object in the linear arrangement of tasks in the task chain, or as a task in parallel with a selected task.

    Choosing the Add as New Task option automatically connects the new object task to the previous object task with the default Success task port. The properties panel for the task chain is also updated with the added objects.

    Choose a task port to connect the first object task to the new object task by dragging from the port to the object task. For more information about working with task ports, see Select Task Ports in a Task Chain.

  4. (Optional) Set auto-retry for a task from the Runtime drop down in the task property panel. This will automatically retry to execute a task in case it fails on the first try.
  5. Continue adding remaining object tasks you want to include in the task chain.

    In addition to adding or replacing object tasks in a task chain, you can drag objects already on the task chain canvas to change the order in which tasks are run. By default, tasks in a task chain are displayed in a vertical direction, from top to bottom. On the left side of the task chain toolbar, the Layout menu provides options to let you change the orientation of tasks from Top to Bottom (Top-Bottom) to Left to Right (Left-Right).

  6. In the properties panel, specify a name for the task chain.

    Task chain properties:

    Properties

    Comments

    Business Name

    Enter a descriptive name to help users identify the object. This name can be changed at any time.

    Technical Name

    Displays the name used in scripts and code, synchronized by default with the Business Name.

    Package

    Select the package to which the object belongs.

    Packages are used to group related objects in order to facilitate their transport between tenants. For more information, see Creating Packages to Export.

    Status

    Displays the deployment status of the task chain: it can be deployed, not deployed, or changes to deploy.

    Objects in Task Chain

    Displays all objects that have been added to the task chain.

    Run Status

    Displays the current run status of the task chain: Not run yet, running, failed, or completed.

    Email Notifications

    Set up email notification for task chain run completion.

    When you click on one of the task chain objects, the properties for this selected task object is displayed in the properties panel:

    Note that you can also access the details of each task chain object in the task chain properties panel. Select the relevant object in the object list and click :

    Task chain object properties:

    Properties

    Comments

    Business Name

    Name of the object

    Technical Name

    Technical name of the object

    Object Type

    A remote table, view, intelligent lookup, data flow, local table, replication flow (load type Initial Only), or transformation flow.

    Activity

    Activity that will be triggered by the task chain.

    Repository Objects:

    • Remote table - Replicate

    • View - Persist

    • Intelligent lookup - Run

    • Data flow - Run

    • Replication Flow - Run

    • Transformation flow - Run

    • Local table:

      • Delete Records with Change Type "Deleted" (for local table where delta capture is enabled)

      • Delete All Records

      • Delete Filtered Records

      For more information, see Controlling Deletion of Local Table Records

    • Local Table (File)

      • Merge: Add, update or delete data into the existing local table (file). Data updates are pushed by a replication flow or SAP BW to the inbound buffer (specific folder in file storage) of a target local table (file). To process data updates from this inbound buffer to the local table (file), and therefore make data visible, a merge task has to run.

      • Optimize:Improve data access performance by optimizing the layout of data in file storage (for example by grouping small files into larger files).

      • Delete Filtered Records: Delete records according to the defined criteria.

      • Delete All Records: Delete records from your local table (file). Records will not be physically deleted but marked as deleted and filtered out when accessing the active records of the local table. They will still consume storage, and they can still be processed by other apps that consume them.

      • Delete Previous Versions (Vacuum): You delete previous versions, which are older than the number of days you have specified. Records that meet your defined criteria will be permanently deleted. The default value is 7 days. The minimum authorized value is 7 so that records from the last 7 days cannot be deleted, and the maximum value is 999 days. In addition, only records that have been fully processed by all delta consumers (for example transformation flows) can be deleted. Note that, however, even if the data is deleted through a vacuum task, data is kept for another 14 days in the file space storage. As it is very likely that you will not be able to go back more than 30 days, we recommend that you do not enter a period longer than 30 days as a vacuum deletion criterion.For more information, see Create a File Space to Load Data in the Object Store

        For more information on local tables (file), see Creating a Local Table (File), Merge or Optimize Your Local Tables (File) and Deleting Local Table (File) Records.

      [Optional]. When you select one of these options, you can override the default Apache Spark Application Settings that were defined at space creation by your administrator.

    • Task Chain - Run

    For remote table and view objects included in a task chain, the default option lets you replicate or persist the data associated with the remote table or view respectively. Or, you can choose to remove the replicated or persisted data.

    • For remote tables, if you choose the Remove Replicated Data option and the remote table object already has data replicated using Snapshot Replication, that table's data will be removed. If the data is being replicated via Real Time Replication, the table's data will also be removed and the object’s data access method will be changed to Remote access. (A log message will be displayed in the remote table log to indicate that the data access type has been changed when the remote table object is run.)

    • For views, if you choose the Remove Persisted Data option, that view's data will be removed.

    Other (Non-Repository) Objects:
    Task Chain Toolbar Objects:

    Apache Spark Settings

    [File Space Only] If you select the activity merge or optimize, you can add additional settings:
    • Use Default: The default application is the application selected in the table settings. If no default application is defined there, the application selected by an administrator during the file space creation is used. However, if the settings have been changed at the object level, in the Data Integration Monitor, this value becomes the default value, overriding the value defined in Workload Management.

    • Define New Setting for This Task: Select another Apache Spark Application that fits your needs.

    • Use a Task Chain Parameter [If an Input Parameter is defined for the task chain] : You can override the Apache Spark Application values by using the values defined in a task chain parameter.

    For more information, see Merge or Optimize Your Local Tables (File), Create Input Parameters in Task Chains and Override the Default Settings to Run Your Transformation Flow (in a File Space).

    Apache Spark Z-Order Columns

    [File Space Only]: When you select Optimize as an activity, you have the option to use Z-ORDER COLUMNS. For more information, see Merge or Optimize Your Local Tables (File)

    Deletion Settings

    Depending on the selected deletion activity, you can add criteria for the records deletion:
    • Define a filter: You define a filter, and based on this filter, the records will be deleted.

    • Define a fixed number: Delete records that are older than the defined value. Minimum value is 0 for local tables and 7 for local tables (file). Maximum value is 999 days.

    • Use a task chain parameter: Take the value defined in the task chain parameter to delete records.

    Status

    Deployment status of the task chain: it can be deployed, not deployed, or changes to deploy

    Ignore Error

    Toggle the Ignore Error button to disregard the status of the task from the calculation of the overall task chain status.

    Input Parameters

    Shows the names of the input parameters in the task object. For more information about input parameters in task chains, see Create Input Parameters in Task Chains.

    Output Parameters

    Shows the names of the output parameters in the task object. For more information about input parameters in task chains, see Create Output Parameters in Task Chains

    Enable Checkup

    [Data Flow Only] If your task running a data flow fails, you can enable this option to perform a health analysis log to be sent to your support team to get help in case of issues. Once the file is generated, go the Flows monitor, and navigate to the details screen to download the run details and send it to your support team. For more information, Monitoring Flows.

    Runtime

    Turn on Enable Auto-Retry for a task in a task chain. This is optional. By default, there is no automatic retry if a task fails. This will automatically retry a task if it helps to help complete a task chain.

    Set from one to three automatic retries for a task if it fails. Choose a period of time, between 15 seconds and 5 minutes, before a task will be run again. This is the minimum amount of 'fail' time between attempts to execute a task. Manually re-running a task chain will pick up from where auto-retry has stopped. This will provide the option of up to 4 attempts to retry a task including the manual rerun of a task chain.

    When you add a shared task chain to the current task chain, the Properties display also lists the space from which the task chain was shared, when that object is selected.

  7. When you’ve finished adding objects to the task chain, save and deploy your new task chain.

    The properties of your task chain are updated.

    After creating and deploying a task chain, you can optionally set up email notification for completion of task chain runs. For more information, see Configure Email Notification for Task Chains and Notification Tasks.

    After you've finished making changes and optionally setting up email notification for the task chain, you can then run the task chain or create a schedule to run your task chain periodically, and navigate to the Task Chains monitor to check your task chain runs. For more information, see Run a Task Chain, Scheduling Data Integration Tasks, and Monitoring Task Chains.

  8. The tools in the editor toolbar help you work with your object throughout its lifecycle:

    Tool

    Description

    (Save)

    Save your changes to the design-time repository. You can use Save As to create a copy of the object.

    See Saving and Deploying Objects.

    (Deploy)

    Deploy your changes to make them available in the run-time environment.

    See Saving and Deploying Objects.

    (Run)

    Run the object to obtain or update its output results.

    You must deploy the object before you can run it.

    (Share)

    Share the object to other spaces.

    See Sharing Entities and Task Chains to Other Spaces.

    (Source Browser)

    Show the Source Browser panel to drag objects into the task chain.

    See Using the Source Browser.

    (Undo) / (Redo)

    Revert the last change to the object or redo a change you have previously undone.

    (Export)

    Export the object to a CSN/JSON file.

    See Exporting Objects to a CSN/JSON File.

    (Impact and Lineage Analysis)

    Open the Impact and Lineage Analysis graph for the object.

    See Impact and Lineage Analysis.

    (Expression Evaluator)

    Open the Expression Evaluator to add an expression object to your task chain.

    See Run Expression Evaluator Tasks in a Task Chain.

    Versions

    Open the Version History dialog for the object.

    See Reviewing and Restoring Object Versions.

    Details

    Toggles the display of the Properties panel.