Metrics for Transformation Flows

View metrics for completed transformation flow runs.

Metrics provide the record count for source and target tables used in the flow. You can view the following metrics for a completed transformation flow run:

  • Apache Spark Application Index

    The index or identifier of the Apache Spark application used to execute the transformation flow. This identifier can be used to track and monitor the Spark application in the cluster's resource manager.

  • Apache Spark Python File Created at Runtime

    Indicates whether a Python file was generated and created at runtime during the Spark-based transformation flow execution. This is common when Python operators are used in the flow.

  • Apache Spark Resource ID

    ID of the job used to run the transformation flow on an Apache Spark runtime. This ID must be provided to the SAP support to retrieve and analyze the jobs, and identify the cause of errors.

  • Average Batch Size (Python)

    Average size of all batches during Python operator execution. It gives a global view of the typical batch size. Comparing this metric against the minimum and maximum batch sizes can help understand the variance in batch sizes. If the average batch size is significantly different from the minimum and maximum, it might suggest inconsistent batch sizing and potential data skew.

  • Batch Column (HANA)

    The name of the column used to partition data into batches for HANA-based transformation flows. This column is used to divide the dataset into logical batches for processing.

  • Batch Size (HANA)

    The number of records processed in each batch during a HANA-based transformation flow run. This metric helps identify the granularity of batch processing and can be used to optimize performance.

  • Compute Engine

    Runtime is used to run the transformation flow. It can be HANA (for a transformation flow in a space with Storage Type SAP HANA Database (disk and In-Memory)) or SPARK (for transformation flow in a space with Storage Type SAP HANA Data Lake Files).

  • Distinct Batch Column Values (excluding NULL) (HANA)

    The count of unique values in the batch column, excluding NULL values. This metric helps understand how many distinct batches will be created during the transformation flow execution on HANA.

  • Flatten Operator Count

    Shows if a Flatten operator is used in the transformation flow. The value is always set to 1 becuase there can't be more than one Flatten operator per transformation flow. See Creating a Flatten Operator

  • HANA Run Mode

    The "Run Mode" is defined in the settings for a transformation flow run: 0 means "Performance-Optimized (Recommended)" and 1 means "Memory-Optimized". For more information about the "Run Mode", see Change Transformation Flow Settings.

  • Incremental Aggregation

    Indicates whether incremental aggregation is enabled for the transformation flow. When enabled, aggregations are computed incrementally, which can improve performance for repeated runs on updated datasets.

  • Load Type

    The load type for a transformation flow run. For more information about load types, see Creating a Transformation Flow.

  • Maximum Batch Size (Python)

    Size of the largest batch processed during Python operator execution. It can help identify if there are unusually large batches, which may indicate data skew (a situation where certain batches contain significantly more data than others). Large batches can lead to performance bottlenecks, as they may take disproportionately longer to process compared to smaller batches.

  • Minimum Batch Size (Python)

    Size of the smallest batch processed during Python operator execution. It provides insight into the smallest unit of data being processed. Consistently small batches might indicate evenly distributed data or very fine-grained task partitioning.

  • Number of Active Records After Run

    The number of active records present in the target table(s) after the transformation flow run completes. This metric is useful for tracking data volume changes and validating the output of the transformation flow.

  • Number of Active Records Before Run

    The number of active records present in the target table(s) before the transformation flow run begins. This metric is useful for comparing data volume before and after the transformation to validate changes.

  • Number of Batches (HANA)

    The total number of batches created and processed during a HANA-based transformation flow run. This metric provides insight into how the data was partitioned and processed.

  • Number of Flagged Records

    The number of records that were flagged during the transformation flow run, typically due to data quality issues, validation failures, or other exceptions. This metric helps monitor data quality and identify problem records.

  • Number of Records

    The number of records written to the target table.

  • Operators with Failed Validation

    The count of operators in the transformation flow that failed validation during the run. This metric helps identify operators that may have configuration issues or data quality problems.

  • Python Operator Count

    The total number of Python operators used in the transformation flow. Python operators allow custom data transformations using Python code.

  • Remote Data Access

    Shows "true" if a remote access has been used when executing the transformation flow. This is possible only when the transformation flow consumes a BW Bridge object imported as remote table.

  • Remove Duplicate Operator Count

    Shows if a Remove Duplicate operator is used in the transformation flow. The value indicates the number of Remove Duplicate operators present in the flow. See Creating a Remove Duplicate Operator.

  • SAP HANA Memory Used

    The peak memory consumption (in mebibytes) for the SAP HANA database while running the transformation flow.

  • Shared Delta Share Source Count

    The number of Delta Sharing source objects consumed by the transformation flow. Delta Sharing sources are external data sources shared through the Delta Sharing protocol.

  • Shared Delta Share Source Volume

    The total data volume from all Delta Sharing source objects consumed during the transformation flow run.

  • Shared Local Table (File) Source Count

    The number of shared local table sources stored in SAP HANA Data Lake Files consumed by the transformation flow.

  • Shared Remote Delta Sharing Source Count

    The number of remote Delta Sharing source objects consumed by the transformation flow. Remote Delta Sharing sources are external data sources accessed through remote connections.

  • Shared SAP HANA Source Count

    The total data volume from all shared SAP HANA source objects consumed during the transformation flow run.

  • Shared SAP HANA Source Volume

    The total data volume from all shared SAP HANA source objects consumed during the transformation flow run.

  • Shared Source

    Indicates whether the transformation flow consumes shared objects as source tables. When "Yes", it means one or more source tables are shared objects rather than local tables.

  • Storage of Target Table

    Target tables created in spaces with SAP HANA Cloud, SAP HANA database storage have the value PAGE if the selected storage is "Disk", or COLUMN if the selected storage is "In-Memory".

  • Total Storage After Run

    The total amount of storage consumed by the target table(s) after the transformation flow run completes. This metric helps monitor storage utilization and plan capacity requirements.

  • Total User-defined Batches (Python)

    Total number of batches processed in the given task. It provides insight into how many times the batch processing logic was ran. A high number of batches might indicate a large dataset, fine-grained batch sizes, or efficient parallel processing..

  • Truncate Target Table before Load

    Indicates whether the Delete All Before Loading option is used for the transformation flow run. For more information, see Create or Add a Target Table to a Transformation Flow.

  • View Transform Count (Graphical)

    The number of view transform operators created using the graphical editor in the transformation flow.

  • View Transform Count (Spark SQL)

    The number of view transform operators created using Spark SQL in the transformation flow.

  • View Transform Count (SQL)

    The number of view transform operators created using SQL statements in the transformation flow.

  • View Transform Count (SQL Script)

    The number of view transform operators created using SQL Script in the transformation flow.