Time Series Data Transfer Engine

Overview

Description

Data transfer engine is mainly built for the back ground operations namely async writing for processed topic writing, re-ingestion runner, ingestion delay runner, and are not directly exposed.

This application provides End-to-End Data Ingestion Delay (End_to_end_Delay) and SAP Leonardo IoT Delay (IoTAE_Ingest_Delay). It also provides the ingested measurement that was used for calculating the ingestion delay.

The following runners are available in Data Transfer Engine:

  • Ingestion Delay Runner – Calculates the ingestion delay. This runner calculates the end-to-end ingestion delay and SAP IoT delay and also reports latest _time value in the sample set of ingested measurements. For more information on how to enable this feature, see Monitoring Time Series Data Ingestion Delay

  • Re-ingestion Runner – In the current Data Ingestion Streaming pipeline, data can fail at various levels with the situation arising re-ingested data to be written at multiple sinks depending on the failure level of the ingestion pipeline.

  • Async Hana Writes Runner – asynchronous writes to HANA tables, thus avoiding the blocked transactions.

Artifacts

This component consists of the following artifacts:

Component Artifacts
Name Description BCP Component
appiot-data-transfer-engine Data transfer engine is mainly built for the background operations namely async writing for processed topics IOT-BSV-BDI
Minimum Hardware and Infrastructure Requirements
Artifact Requirements
appiot-data-transfer-engine Memory 2GB

Documentation

You may find further useful information in the following resources:

Operations

Availability Checks

The following table gives you an Overview of the general purpose of the various availability checks, followed by a list of links that point directly to the check results for the various artifacts and locations:

General Overview of the Purpose of Availability Checks
Check Purpose
L3 Direct ping against the application. Shows whether the application is up and running.
L3+ Ping against the application via the app router. Can reveal app router-related problems.
L4 Requests some data from the application via the app router. Involves HANA interaction. Shows issues with the SAP HANA database.
L5 Shows problems with tenant onboarding in the business partner during the last hour.
Check Results
Artifact Landscape URL
appiot-data-transfer-engine EU10

L3: Monitor

L3+: URL

L4: URL

L5: URL

appiot-data-transfer-engine US10

L3: Monitor

L3+: URL

L4: URL

L5: URL

Kibana

Authorization (Cloud Access Manager (CAM) profile): CP IoTAE MoD/DoD Auditor Production

From the Kibana dashboard, set up a filter based on a condition like the following example:

{
  "query": {
    "wildcard": {
      "component_name": {
        "value": "appiot-data-transfer*"
      }
    }
  }
}
Kibana Entry Points
Landscape Entry Point
EU10

Dashboard Start Page

HTTP 5xx Errors

US10

Dashboard Start Page

HTTP 5xx Errors

Dynatrace

Dynatrace Performance Monitor

Dynatrace Performance Monitor
Landscapte Entry Point
EU10

Dashboard Start Page

US10

Dashboard Start Page

Grafana

Grafana Database Monitor
Landscapte Entry Point
EU10

Monitor

US10

Monitor

Application Logs Available to Customers

n/a

Miscellaneous

  • Spark Master UI dashboard information provided in separate Spark UI section

  • Data Ingestion Dashboard

Troubleshooting

Availability

Recommended Actions for Availability Errors
Check Root Cause Solution
L3

IoTAE_AWS-Live-EU10_appiot-data-transfer-engine_L3

 

Go to SAP Cloud Platform Cockpit

  1. Search for the application appiot-data-transfer-engine.

  2. Check if the status of this application is started.

  3. If the status is started, trigger an HTTP GET request to the ping API https://appiot-data-transfer-engine.<<domain>>/Ping and the expected response is Pong.

    • If the response is Pong, the application is up and running. Wait for another two minutes and check if IoTAE_AWS-Live-EU10_appiot-data-transfer-engine_L3 is still down. Report the issue to the Cloud Engineer on Duty considering there could be an issue in SAP Cloud Platform (SCP) Availability Service.

    • If the response is not Pong, wait for two minutes and check if the availability check status is Up again.

      If the two minute wait check fails and the IoTAE_AWS-Live-EU10_appiot-data-transfer-engine_L3 check continues to be down, restart the appiot-data-transfer-engine from SCP Cloud Cockpit.

  4. Wait for five minutes for the application to get successfully restarted. Repeat step (2) through (3) for confirmation.

  5. If the IoTAE_AWS-Live-EU10_appiot-data-transfer-engine_L3 check continues to be down, inform the Cloud Engineer on Duty that SAP IoT service is down.

HTTP Errors

The following table presents a list of error-related HTTP status codes, the typical root causes, and the recommended actions for solving the issue.

Recommended Actions for Solving HTTP Errors
Status Code Root Cause Solution
400 n/a n/a
401 n/a n/a
403 n/a n/a
404 n/a n/a
429 n/a n/a
500 n/a n/a
502 n/a n/a
902 n/a n/a

General Metrics

The following table presents a list of common metrics for this component with a threshold value indicating critical system load, along with a recommended action as well as the preferred monitoring tool, where applicable.

General Component Metrics
Metric Tool Threshold Action
CPU Usage Dynatrace n/a n/a
Memory Usage Dynatrace n/a n/a
No. of Requests/Time Dynatrace n/a n/a
Avg. Response Time Dynatrace n/a n/a
Incoming Byte Rate n/a n/a n/a
Outgoing Byte Rate n/a n/a n/a

Additional Metrics

n/a

Miscellaneous

n/a

Issue Handling

How to validate if ingestion delay metrics are retrieved correctly

  1. Read the value of the field LastBusinessTimestampProcessed from response payload obtained by triggering the API that retrieves the ingestion delay for the tenant (see the DoD Tasks section for the API details).

  2. Open the Spark Master UI. Choose the context ID of the default job. In the following screen, choose the Streaming tab and identify the batch based on the LastBusinessTimestampProcessed value. This gives the batch time in which the record was processed by Spark Job.

  3. The sum of difference between the time in step (1) and step (2) and the delta time spent in computation inside Data Transfer Engine gives the approximate range for considering End_to_end_Delay.

  4. The value for IoTAE_Ingest_Delay could be either equal to or little lesser than the value obtained in step (2).

  5. A value less than or equal to 10 seconds for IoTAE_Ingest_Delay holds good.

Where to check if the GET API for ingestion delay returns an empty result?

  1. Check if ingestion is stopped for the given tenant

    • Open Grafana dashboard and check for the time when the message was received last for the <tenant-name>.sap.v1.iot.processed_timeseries.

  2. Check if the entry for the IngestionSamplingRate column is deleted from the sap.crossTenant.p::Tenant.Tenant table.
    • As the column is deleted, the tenant is not considered for monitoring the data ingestion delay.

    • Log in to HANA-SHARED-HDI and execute the query select "Tenant", "IngestionSamplingRate" from "<<schema>>"."sap.crossTenant.p::Tenant.Tenant" where "Tenant" = '<tenant-name>'.

  3. Check if the ingested data is written into the error store
    • Use sensorAlternateId and CapabilityAlternateId to derive the sourceId, and accordingly identify the structureID. Using these details, check the Cassandra error-message table for the messages against the provided sensor or capability. You can execute the query select * from iotae_dataingestion.errormessage where tenant ='<<tenant name>>' and epoch_day=<<epoch day for which error occurred>>;.

Where to check If End_to_end_Delay holds very large value and if there is a huge difference between End_to_end_Delay and IoTAE_Ingest_Delay?

  1. Open Spark UI and check if the following occurs:

    • Spark Job has queued up batches

    • Spark Job is restarted and have huge load with initial batches having scheduling delay in minutes

DoD Tasks

Monitoring

This section explains a list of APIs that a DoD can use to monitor the health of different applications.

  • Retrieve the list of tenants configured for monitoring

    This GET API retrieves a list of tenants that are configured for the data transfer engine application to calculate the ingestion delay.

    Request URL: https://<monitoring-dataingestion-sap>/api/dte/ingestion-delay-monitor/polling-interval/configured-tenants

    Header parameter: Authorization <<bearer token>>

    Response Status Code: 200 OK

    Response Body:
    {
        "Interval-poll-time": [
            "tenant-name"
        ],
    
        "Interval-poll-time ": [
            "tenant-name"
        ],
    
        " Interval-poll-time": [
            "tenant-name",
        ]
    }
    
  • Retrieve the list of things configured for the tenant

    This GET API retrieves the list of things that have been created for the data transfer engine application. By default, the ingestion delay is fetched for 1 hour. You can use the query parameter timerange to specify the duration for which the ingestion delay needs to be fetched.

    Request URL: https://<monitoring-dataingestion-sap>/api/dte/ingestion-delay-monitor/configured-thing/tenants/<tenant-name>

    Response Status Code: 200 OK

    Response Body
    {
    "Thing": "<thingId>"
    	       }
    
  • Retrieve the ingestion delay for the tenant

    This GET API retrieves the ingestion delay for the tenant that is provided in the request URL.

    Request URL: https://<monitoring-dataingestion-sap>/api/dte/ingestion-delay-monitor/configured-thing/tenants/<tenant-name>

    Response Status Code: 200 OK

    Response Body
    "tenant": "<tenant-name>",
    "ingestionDelayInfoList": [
    {
    "lastBusinessTimestampProcessed": "1550131000686",
    "endToEndDelay": "43",
    "iotaeIngestDelay": "10575"
    }
    
  • Retrieve ingestion delay for all tenants

    This GET API retrieves the ingestion delay for all the tenants.

    Request URL: https://<monitoring-dataingestion-sap>/api/dte/ingestion-delay-monitor/configured-thing/tenants

    Response Status Code: 200 OK

    Response Body:
    [
        {
            "tenant": "<tenant1>",
            "thing": "",
            "delayInfos": [
    {
                    "lastBusinessTimestampProcessed": "1583168378000",
                    "endToEndDelay": "13066",
                    "iotaeIngestDelay": "3297"
                },
             ]
        },
        {
            "tenant": "<tenant2>",
            "thing": "",
            "delayInfos": [
    {
                    "lastBusinessTimestampProcessed": "1583168378000",
                    "endToEndDelay": "13215",
                    "iotaeIngestDelay": "3440"
                }]
        },
        {
            "tenant": ""<tenant3>",
            "thing": "",
            "delayInfos": [
    {
                    "lastBusinessTimestampProcessed": "1583168184000",
                    "endToEndDelay": "7206",
                    "iotaeIngestDelay": "1466"
                }]
        }
    ]
    

Async HANA Writes Runner (ASYNC_DATAEXISTENCE_RUNNER)

  1. After deployment, ensure that the ASYNC_DATAEXISTENCE_RUNNER thread is started. To ensure that the thread is running, check if the following is printed in the application logs:

    Runner created successfully:: ASYNC_DATAEXISTENCE_RUNNER

  2. If the runner is not up and running the re-try runner within it keeps retrying to bring it up. To ensure ASYNC_DATAEXISTENCE_RUNNER retry runner thread is started, check if the following is printed in the application logs after deployment:

    Runner created successfully:: ASYNC_DATAEXISTENCE_RETRY_RUNNER

  3. Topic and consumer group of the two runners are:

    • DataExistence Runner

      Topic: sap.iot.timeseries.dataexistence.v1

      Consumer-group: asyncde-runner

    • DataExistenceRetry Runner

      Topic: sap.iot.timeseries.dataexistence_error.v1

      Consumer-group: asyncde-errorwriter-runner

Re-ingestion Runner (REINGESTION_RUNNER)

  1. The REINGESTION_RUNNER thread is run once daily in the live landscape.

  2. After deployment, ensure that the REINGESTION_RUNNER thread is started. To ensure that the thread is running, check if the following is printed in the application logs:

    Runner created successfully:: REINGESTION_RUNNER

  3. Topic and consumer group id is:

    Topic: errormessage.sap.iot.timeseries

    Group ID: kafka-reingestion-runner

  4. In every scheduled run, snapshot of the error topic to be scanned is logged. Search for the log "Offset to be read from". This also prints the partition information and latest message present in the topic at the time of snapshot.

  5. After successful completion of one scheduled run of the re-ingestion runner, check for the log "Closing consumer".