Time Series Data Transfer Engine
Overview
Description
Data transfer engine is mainly built for the back ground operations namely async writing for processed topic writing, re-ingestion runner, ingestion delay runner, and are not directly exposed.
This application provides End-to-End Data Ingestion Delay (End_to_end_Delay) and SAP Leonardo IoT Delay (IoTAE_Ingest_Delay). It also provides the ingested measurement that was used for calculating the ingestion delay.
The following runners are available in Data Transfer Engine:
-
Ingestion Delay Runner – Calculates the ingestion delay. This runner calculates the end-to-end ingestion delay and SAP IoT delay and also reports latest _time value in the sample set of ingested measurements. For more information on how to enable this feature, see Monitoring Time Series Data Ingestion Delay
-
Re-ingestion Runner – In the current Data Ingestion Streaming pipeline, data can fail at various levels with the situation arising re-ingested data to be written at multiple sinks depending on the failure level of the ingestion pipeline.
-
Async Hana Writes Runner – asynchronous writes to HANA tables, thus avoiding the blocked transactions.
Artifacts
This component consists of the following artifacts:
| Name | Description | BCP Component |
|---|---|---|
| appiot-data-transfer-engine | Data transfer engine is mainly built for the background operations namely async writing for processed topics | IOT-BSV-BDI |
| Artifact | Requirements |
|---|---|
| appiot-data-transfer-engine | Memory 2GB |
Documentation
You may find further useful information in the following resources:
- SAP Help Portal:
- SAP API Business Hub
- n/a
Operations
Availability Checks
The following table gives you an Overview of the general purpose of the various availability checks, followed by a list of links that point directly to the check results for the various artifacts and locations:
| Check | Purpose |
|---|---|
| L3 | Direct ping against the application. Shows whether the application is up and running. |
| L3+ | Ping against the application via the app router. Can reveal app router-related problems. |
| L4 | Requests some data from the application via the app router. Involves HANA interaction. Shows issues with the SAP HANA database. |
| L5 | Shows problems with tenant onboarding in the business partner during the last hour. |
Kibana
Authorization (Cloud Access Manager (CAM) profile): CP IoTAE MoD/DoD Auditor Production
From the Kibana dashboard, set up a filter based on a condition like the following example:
{
"query": {
"wildcard": {
"component_name": {
"value": "appiot-data-transfer*"
}
}
}
}
| Landscape | Entry Point |
|---|---|
| EU10 | |
| US10 |
Dynatrace
Dynatrace Performance Monitor
| Landscapte | Entry Point |
|---|---|
| EU10 | |
| US10 |
Application Logs Available to Customers
n/a
Miscellaneous
-
Spark Master UI dashboard information provided in separate Spark UI section
-
Data Ingestion Dashboard
-
Summary on Coldstore Writer
-
Offset lags by partition
-
Troubleshooting
Availability
| Check | Root Cause | Solution |
|---|---|---|
| L3
IoTAE_AWS-Live-EU10_appiot-data-transfer-engine_L3 |
Go to SAP Cloud Platform Cockpit
|
HTTP Errors
The following table presents a list of error-related HTTP status codes, the typical root causes, and the recommended actions for solving the issue.
| Status Code | Root Cause | Solution |
|---|---|---|
| 400 | n/a | n/a |
| 401 | n/a | n/a |
| 403 | n/a | n/a |
| 404 | n/a | n/a |
| 429 | n/a | n/a |
| 500 | n/a | n/a |
| 502 | n/a | n/a |
| 902 | n/a | n/a |
General Metrics
The following table presents a list of common metrics for this component with a threshold value indicating critical system load, along with a recommended action as well as the preferred monitoring tool, where applicable.
| Metric | Tool | Threshold | Action |
|---|---|---|---|
| CPU Usage | Dynatrace | n/a | n/a |
| Memory Usage | Dynatrace | n/a | n/a |
| No. of Requests/Time | Dynatrace | n/a | n/a |
| Avg. Response Time | Dynatrace | n/a | n/a |
| Incoming Byte Rate | n/a | n/a | n/a |
| Outgoing Byte Rate | n/a | n/a | n/a |
Additional Metrics
n/a
Miscellaneous
n/a
Issue Handling
How to validate if ingestion delay metrics are retrieved correctly
-
Read the value of the field LastBusinessTimestampProcessed from response payload obtained by triggering the API that retrieves the ingestion delay for the tenant (see the DoD Tasks section for the API details).
-
Open the Spark Master UI. Choose the context ID of the default job. In the following screen, choose the Streaming tab and identify the batch based on the LastBusinessTimestampProcessed value. This gives the batch time in which the record was processed by Spark Job.
-
The sum of difference between the time in step (1) and step (2) and the delta time spent in computation inside Data Transfer Engine gives the approximate range for considering End_to_end_Delay.
-
The value for IoTAE_Ingest_Delay could be either equal to or little lesser than the value obtained in step (2).
-
A value less than or equal to 10 seconds for IoTAE_Ingest_Delay holds good.
Where to check if the GET API for ingestion delay returns an empty result?
-
Check if ingestion is stopped for the given tenant
-
Open Grafana dashboard and check for the time when the message was received last for the <tenant-name>.sap.v1.iot.processed_timeseries.
-
- Check if the entry for the IngestionSamplingRate column is deleted from the sap.crossTenant.p::Tenant.Tenant table.
-
As the column is deleted, the tenant is not considered for monitoring the data ingestion delay.
-
Log in to HANA-SHARED-HDI and execute the query select "Tenant", "IngestionSamplingRate" from "<<schema>>"."sap.crossTenant.p::Tenant.Tenant" where "Tenant" = '<tenant-name>'.
-
- Check if the ingested data is written into the error store
-
Use sensorAlternateId and CapabilityAlternateId to derive the sourceId, and accordingly identify the structureID. Using these details, check the Cassandra error-message table for the messages against the provided sensor or capability. You can execute the query select * from iotae_dataingestion.errormessage where tenant ='<<tenant name>>' and epoch_day=<<epoch day for which error occurred>>;.
-
Where to check If End_to_end_Delay holds very large value and if there is a huge difference between End_to_end_Delay and IoTAE_Ingest_Delay?
-
Open Spark UI and check if the following occurs:
-
Spark Job has queued up batches
-
Spark Job is restarted and have huge load with initial batches having scheduling delay in minutes
-
DoD Tasks
Monitoring
This section explains a list of APIs that a DoD can use to monitor the health of different applications.
-
Retrieve the list of tenants configured for monitoring
This GET API retrieves a list of tenants that are configured for the data transfer engine application to calculate the ingestion delay.
Request URL: https://<monitoring-dataingestion-sap>/api/dte/ingestion-delay-monitor/polling-interval/configured-tenants
Header parameter: Authorization <<bearer token>>
Response Status Code: 200 OK
Response Body:{ "Interval-poll-time": [ "tenant-name" ], "Interval-poll-time ": [ "tenant-name" ], " Interval-poll-time": [ "tenant-name", ] } -
Retrieve the list of things configured for the tenant
This GET API retrieves the list of things that have been created for the data transfer engine application. By default, the ingestion delay is fetched for 1 hour. You can use the query parameter timerange to specify the duration for which the ingestion delay needs to be fetched.
Request URL: https://<monitoring-dataingestion-sap>/api/dte/ingestion-delay-monitor/configured-thing/tenants/<tenant-name>
Response Status Code: 200 OK
Response Body{ "Thing": "<thingId>" } -
Retrieve the ingestion delay for the tenant
This GET API retrieves the ingestion delay for the tenant that is provided in the request URL.
Request URL: https://<monitoring-dataingestion-sap>/api/dte/ingestion-delay-monitor/configured-thing/tenants/<tenant-name>
Response Status Code: 200 OK
Response Body"tenant": "<tenant-name>", "ingestionDelayInfoList": [ { "lastBusinessTimestampProcessed": "1550131000686", "endToEndDelay": "43", "iotaeIngestDelay": "10575" } -
Retrieve ingestion delay for all tenants
This GET API retrieves the ingestion delay for all the tenants.
Request URL: https://<monitoring-dataingestion-sap>/api/dte/ingestion-delay-monitor/configured-thing/tenants
Response Status Code: 200 OK
Response Body:[ { "tenant": "<tenant1>", "thing": "", "delayInfos": [ { "lastBusinessTimestampProcessed": "1583168378000", "endToEndDelay": "13066", "iotaeIngestDelay": "3297" }, ] }, { "tenant": "<tenant2>", "thing": "", "delayInfos": [ { "lastBusinessTimestampProcessed": "1583168378000", "endToEndDelay": "13215", "iotaeIngestDelay": "3440" }] }, { "tenant": ""<tenant3>", "thing": "", "delayInfos": [ { "lastBusinessTimestampProcessed": "1583168184000", "endToEndDelay": "7206", "iotaeIngestDelay": "1466" }] } ]
Async HANA Writes Runner (ASYNC_DATAEXISTENCE_RUNNER)
-
After deployment, ensure that the ASYNC_DATAEXISTENCE_RUNNER thread is started. To ensure that the thread is running, check if the following is printed in the application logs:
Runner created successfully:: ASYNC_DATAEXISTENCE_RUNNER
-
If the runner is not up and running the re-try runner within it keeps retrying to bring it up. To ensure ASYNC_DATAEXISTENCE_RUNNER retry runner thread is started, check if the following is printed in the application logs after deployment:
Runner created successfully:: ASYNC_DATAEXISTENCE_RETRY_RUNNER
-
Topic and consumer group of the two runners are:
-
DataExistence Runner
Topic: sap.iot.timeseries.dataexistence.v1
Consumer-group: asyncde-runner
-
DataExistenceRetry Runner
Topic: sap.iot.timeseries.dataexistence_error.v1
Consumer-group: asyncde-errorwriter-runner
-
Re-ingestion Runner (REINGESTION_RUNNER)
-
The REINGESTION_RUNNER thread is run once daily in the live landscape.
-
After deployment, ensure that the REINGESTION_RUNNER thread is started. To ensure that the thread is running, check if the following is printed in the application logs:
Runner created successfully:: REINGESTION_RUNNER
-
Topic and consumer group id is:
Topic: errormessage.sap.iot.timeseries
Group ID: kafka-reingestion-runner
-
In every scheduled run, snapshot of the error topic to be scanned is logged. Search for the log "Offset to be read from". This also prints the partition information and latest message present in the topic at the time of snapshot.
-
After successful completion of one scheduled run of the re-ingestion runner, check for the log "Closing consumer".