Creating a Python Operator

Insert a Python operator to transform incoming data with a Python script and output structured data to the next operator.

In a transformation flow in a file space, the Python operator receives the data from a previous operator. You can provide transformation logic as a body of the transform function in the Script property of the operator. Incoming data is fed into the data parameter of the transform function and the result from this function is returned to the output.

The Python operator allows data manipulation and vector operations by providing support for builtins (version 3.11), NumPy (version 1.26.4), and Pandas (version 2.2.2) modules. Non-I/O objects and functions of NumPy and Pandas can be used with aliases np and pd, respectively, without any requirements to explicitly import them.

The incoming data parameter in the transform function is of type Pandas DataFrame. The input table is converted into a DataFrame and fed into transform function as data parameter. You are expected to provide scripts for the intended transformation of the incoming DataFrame and also return a valid DataFrame from transform function. It is important that the returning DataFrame from the transform function has the same column names, types and order as the specified table for the output. Otherwise, execution of the data flow results in failure.

  1. Drag and drop an object onto the source operator or select the View Transform to display its context menu, and click Python operator to create the Python operator.

  2. Click the Python operator to display its properties in the side panel.

  3. In the General section, under Input Mode, select your operator processing settings:

    • System-Managed: [default] The operator uses mapInPandas for parallel batch-wise processing. To ensure efficient and accurate data transformation, avoid performing global aggregation, grouping, or similar operations inside the Python code, as these require full dataset context. Instead, either use the View Transform operator for aggregation and grouping, or split the transformation flow to isolate operations requiring global context.

    • User-Defined Batches: The operator uses applyInPandas. Defining batches based on specific column values enables more control over data processing. You can organize and analyze data more efficiently by grouping related records together. Batches should contain similar numbers of records to help avoid out of memory issues. To define batches, click Batch Columns and select one to three batch columns.

    A user-defined batch in Python operator is a subset of data logically grouped when running the Python code. However, unlike regular batches,  it doesn't control how the data is read from the source. Ensure all data for a user-defined batch exists in the object store before the Python operator processes it.

  4. [optional] Resolve errors in your data records. You can:

    • Enable Support Data Validation to automatically track errors and manage records that fail validation. It allows you to improve data quality and reprocess erroneous records without manual intervention. A local table for error tracking is automatically generated and is SAP-Managed, read-only, and named after your transformation flow with the suffix _python1_remediation(.). You can then mass-correct the erroneous data in this table using Python.

    • Use the Data Remediation table to manually correct erroneous records in the Data Integration monitor.

    See Resolving Erroneous Records Using the Data Remediation Table.

  5. In the Script section, enter your Python script to transform the incoming data and produce an output schema.

    For information about Python support in the Python operator, see Python Operator Reference.

  6. In the Columns section, manually specify the columns that will be output by your Python operator.