Raw Data Outbound Logs API
Introduction
The SAP Cloud ALM Raw Data Outbound Logs API is used to read logs produced by the different SAP Cloud ALM monitoring use-cases with the OTeL format in PULL mode. This API can be used to integrate data from an open source, in-house tool or third-party tool by exporting SAP Cloud ALM data in the OTel format.
The number of log records returned by the API depends on the time period and time filter parameters used during the API calls.
The following metrics data providers are supported:
|
Log Providers |
SAP Cloud ALM Monitoring Use Cases |
|---|---|
|
exm.im |
Exception Monitoring for Integration Monitoring |
|
exm.jam |
Exception Monitoring for Job Monitoring |
|
bsm |
Business Services Management (status events) |
|
im |
Integration Monitoring |
|
iep |
Monitoring Events (see prerequisites) |
|
jam.def |
Job Automation Monitoring Definition Logs |
|
jam.run |
Job Automation Monitoring Runtime Logs |
|
jam.run_step |
Job Automation Monitoring Runtime Step Logs |
Data Protection and Privacy
The SAP Cloud ALM API Raw Data Outbound Logs can return personal and sensitive data which must not be accessed or stored without technical and organizational measures to safeguard data protection and privacy.
Check the following guide for more information: https://help.sap.com/docs/cloud-alm/setup-administration/data-protection-privacy?locale=en-US
Type
Pull API
URL Format
-
https://<tenant name>.<provider name>/api/calm-logs/v1/logs
Documentation
Log data are retrieved from SAP Cloud ALM via an HTTP API end-point documented in the SAP API HUB.
Operations
-
GET /logs
-
calm-logs/v1/logs GET
-
Authorizations
|
Security Scope |
Description |
|---|---|
|
calm-api.logs.read |
Mandatory security scope to access any CALM logs |
|
calm-api.exm.read |
To access Exception Monitoring use case data |
|
calm-api.exm.personal.read |
To read log in Exception Monitoring use case. This security scope is necessary to allow extraction of data that could contain personal and sensitive data. |
|
calm-api.bsm.read |
To get access to Business Services Management data |
|
calm-api.integration-monitoring.read |
To get access to the Integration Monitoring data |
|
calm-api.integration-monitoring.personal.read |
To read log in Integration Monitoring use case. This security scope is necessary to allow extraction of data that could contain personal and sensitive data. |
|
calm-api.iep.read |
To get access to the Monitoring Event data |
|
calm-api.jobs.read |
To get access to the Jobs & Automation Monitoring data |
|
calm-api.jobs.personal.read |
To get access to the Jobs & Automation Monitoring personal data |
A request without security scope authorisation always receives an authorisation exception (HTTP 403).
Examples
-
https://{{CALM_URL}}/api/calm-logs/v1/logs?version=V1&period=C1H&provider=exm.im&serviceId=3c8c7cd1-a346-4af1-a52e-428e53848669&category=ABAP Job Application Log&format=protobuf-binary
-
https://{{CALM_URL}}/api/calm-logs/v1/logs?provider=im&serviceId=93794776-c9a3-4b80-8532-38d408628bd9,93794776-c9a3-4b80-8532-38d408628bd9&category=SAP Integration Suite Messages,Concur Objects,PI Messages,IDoc&format=protobuf-json&period=60M&limit=100&onLimit=TRUNCATE
API Details
The SAP Cloud ALM Raw Data Outbound Logs request the following parameters:
Query parameters
|
Name |
Description |
Comments |
|---|---|---|
|
format |
Expected output format of the logs api. You can select between text, protobuf-binary or protobuf-json format.
Supported Values:
|
String Default: protobuf-binary |
|
provider |
The log data provider accessed though the logs api.
Supported values:
|
String Mandatory |
|
version |
Version of the logs provider. |
String Example: v1 Default: Oldest non-deprecated version. |
|
period |
Semantic period to collect. This parameter can be used instead of parameters “from” and “to”; it is converted to “from” and “to”. Parameter “to” is set to Now, where 'Now' is the current UTC unix time on SAP Cloud ALM side, and parameter “from” is calculated by subtracting “period” to parameter “to”. If parameters “to” and/or “from” are provided in addition to “period”, then an error is returned (incompatible parameters “from”/”to” and “period”). Possible values: <X>M where X is in [1,60] unless provider authorizes larger from/to period (e.g. bsm has no restriction for requested from/to period)..
|
String Default: 5M |
|
from |
UTC timestamp indicating the beginning of the collection period. The largest possible period to collect is 1 hour; if the period (from/to) goes above one hour, then an error is returned. Caution: “from” timestamp is inclusive. |
Optional Format: YYYYMMDDHHmmss |
|
to |
UTC timestamp indicating the end of the collection period. The oldest timestamp accepted for parameter “to” is defined by the Logs Provider, if “to” goes beyond this limit an error should be returned by the Logs Provider. If “to” is not provided or if “to” value is in the future, then the newest log records until Now are returned. Caution: “to” is exclusive. |
Optional Format: YYYYMMDDHHmmss Default: Now |
|
observedTimestamp |
Boolean. Indicates if the "from" & "to" timestamps correspond to measured timestamps (observedTimestamp = false) or observed timestamps (observedTimestamp = true). Since a log measured at time t1 might be observed later at t1+delta (delta time is due to data collection infrastructure), it may be interesting to rely on observed timestamps which guarantee the presence of the logs. Not relevant for providers bsm since they doesn't distinguish between timestamp and observed timestamp. For provider jam.run, observedTimestamp=true is the default and the only mode supported. For provider jam.def, period (from/to) is not used, all job definitions are always returned. |
Default: false |
|
<filters> |
Log Records filters are passed as URL parameter. The list of filters depends on the Logs Data Provider. See next chapter for a description of filters per logs data providers. |
Optional |
|
limit |
Pagination support. Maximum number of log records to return. Minimum value is 1, maximum value is 1000. |
Default: 100 |
|
offset |
Pagination support. Number of log records to skip. If the offset goes beyond the total number of log records available, then no log records are returned (no error). |
Default: 0 |
|
onLimit |
Indicates what to do if the number of records goes above the maximum limit (parameter “limit”). Possible values are: ERROR (return an error), TRUNCATE: truncate the response. |
Default: ERROR |
Filters
Exception Monitoring
|
Filter Name |
Description |
Comment |
|---|---|---|
|
serviceId |
Comma-separated list of LMS service Ids. Minimum 1 service id, maximum 30 service ids. |
Mandatory |
|
category |
Exception source category |
Optional |
|
status |
Resolved, Warning or Error |
Optional |
|
message |
Wildcard '*' can be used to support following scenarios:
|
Optional |
Business Service Management
Remark: only the log corresponding to the last event status is returned and not the complete action logs. For example, if an event has been announced, updated and completed today and we request the BSM logs for today, we get only the log "event completed".
|
Filter Name |
Description |
Comment |
|---|---|---|
|
serviceId |
LMS service Id |
Mandatory. Can be for business service, cloud service or system |
|
from |
Timestamp indicating the beginning of the collection period. |
from and to indicate the time when the event was updated and not necessarily the time when the event starts or stops For example:
BSM OTEL API does not distinguish between observedTimestamp and measured timestamp There is no restriction on period length and on how far we can go in the past. Only restriction is from <= to. |
|
to |
Timestamp indicating the end of the collection period. |
from and to indicate the time when the event was updated and not necessarily the time when the event starts or stops For example:
BSM OTEL API does not distinguish between observedTimestamp and measured timestamp There is no restriction on period length and on how far we can go in the past. Only restriction is from <= to. |
|
eventDomain |
Event type for which the client needs to get events. Available values: Maintenance, Degradation, Disruption, Communication |
Optional |
|
eventPhase |
Phase of the event . Available values: Announcement, Detection, Status Update, Start, Completion, Resolution, Root Cause Analysis. |
Optional |
|
showDescription |
Whether the description should be returned or not . Possible values X or <space>. Default <space> |
|
|
eventSource |
source of the event. possible values: SAP, Customer, API or External (in the future ) |
Integration Monitoring
|
Filter Name |
Description |
Comment |
|---|---|---|
|
serviceId |
Comma-separated list of LMS service Ids |
Mandatory |
|
from / to |
Timestamp indicating the beginning and the end of the collection period. All messages with start time between from and to timestamps will be returned by the interface. |
Mandatory from/to or period |
|
category |
Comma-separated list of Category labels as displayed in Integration Monitoring UI |
Mandatory |
Event Monitoring
|
Filter Name |
Description |
Comment |
|---|---|---|
|
serviceId |
LMS service Id |
Mandatory |
|
from / to |
Timestamp indicating the beginning and the end of the collection period. All messages with start time between from and to timestamps will be returned by the interface. |
Mandatory |
|
usecase |
Monitoring usecase Allowed values: "CALM.EXM", "CALM.IM", "CALM.BPMON", "CALM.BSM", "CALM.HM", "CALM.JM", "CALM.SUM" |
Optional Only one value |
|
eventsubtype |
Metric technical name Ex: "calm.im.cpi.msg" |
Optional Only one value |
Jobs & Automation Monitoring
|
Filter Name |
Description |
Comment |
|---|---|---|
|
serviceId |
Comma-separated list of LMS service Ids. Minimum 1 service id, maximum 27 service ids. |
Mandatory |
|
from / to |
Timestamp indicating the beginning and the end of the collection period. All messages with start time between from and to timestamps will be returned by the interface. |
Mandatory from / to or period |
|
jobType |
Job type CF_DM_PROCESS → SAP DM Production Process HCP_IPS_JOB → SAP Cloud Identity Provisioning Job IRPA → SAP Intelligent RPA Job NW_ABAP_JOB → SAP ABAP Job NW_ABAP_PC → SAP BW Process Chain NW_ABAP_WF → SAP Business Workflow NW_APPL_JOB → SAP Application Job SPA_AUTOMATION → SAP Build Automation Job SPA_PROCESS → SAP Build Process |
Optional |
|
jobName |
Job name |
Optional |
|
jobContext |
Job context |
Optional |
|
name |
Step name |
Optional. Relevant with provider jam.run_step |
Status Codes
|
Status Code |
Description |
|---|---|
|
202 |
Data accepted |
|
400 |
Structure of the request is invalid. Unknown parameter “XYZ” Parameters “from”/”to” and “period” are incompatible. Parameter XYZ has an unsupported value “XYZ” |
|
403 |
Authentication failure. Security scope is missing |
|
404 |
The request path is incorrect. Unknown use-case "XYZ" Unknown use-case version "X" Service "XYZ" not found |
|
408 |
The request took too long to reach the endpoint. |
|
413 |
The payload was too big. |
|
428 |
Pre-condition not full filled. |
|
429 |
The request rate quota has been exceeded. |
|
500 |
Internal Server Error |
|
502 |
Bad Gateway |
|
503 |
Service unavailable |
|
504 |
Gateway Timeout |
Response headers
|
Header |
Description |
|---|---|
|
x-total-count |
Total number of log records. Returned only if parameter "onLimit" = "TRUNCATE" |
Response Body Example
|
EXM - Response Body Example |
|---|
|
|
|
BSM - Response Body Example |
|---|
|
Monitoring Event - Response body example |
|---|
Limitations
-
Throughput:
-
Maximum 100 requests per minute
-
-
Limits:
-
Maximum log records per response: 1000 (HTTP Error Code 403 + Error Message)
-
Prerequisites
Before to access the Raw Data Outbound Logs API for Monitoring Event, you must enable the action "Store Event Payload for 24 Hours" on the required events. More details are available in Intelligent Event Processing documentation here.
Restrictions
The json protobuf encoding does not support the deviations described https://opentelemetry.io/docs/specs/otlp/#json-protobuf-encoding
; to manage correlation id use the protobuf binary format. With protobuf json format, spanId and traceId are base-64-encoded and not case-insensitive-hex-encoded-strings.
Models mapping
-
Some mapping rules are common to all scenarios (see section Common mapping).
-
Some mapping rules are specific to each scenario (see dedicated scenario mapping tables).
|
SAP Cloud ALM (source) |
OTLP (target) |
Comments |
|---|---|---|
|
Service type |
resourceLogs / resource / attribute / service.namespace |
|
|
Service name |
resourceLogs / resource / attribute / service.name |
|
|
Service description |
resourceLogs / resource / attribute / sap.service.display_name |
|
|
Service attributes |
list of key value-pairs resources attributes |
Depends on data source |
|
Logs data provider's version |
resourceLogs / scopeLogs / scope / version |
Depends on data source |
|
Logs data provider's technical name |
resourceLogs / scopeLogs / scope / name |
Depends on data source |
|
resourceLogs / resource / attribute / service.instance.id |
|
|
resourceLogs / resource / attribute / telemetry.sdk.name |
Depends on data source |
|
|
resourceLogs / resource / attribute / telemetry.sdk.language |
Depends on data source |
|
|
resourceLogs / resource / attribute / telemetry.sdk.version |
Depends on data source |
Exception Monitoring Mapping
In addition to the common mapping rules, the table below describes the mapping rules specific to Exception Monitoring scenario.
|
Exception Monitoring |
OTLP LogRecord |
Comment |
|---|---|---|
|
Message |
logRecord.body |
|
|
Status |
logRecord.severityNumber / logRecord.severityText |
|
|
Timestamp |
logRecord.timeUnixNano |
|
|
Saved_timestamp |
logRecord.observedTimeUnixNano |
|
|
Association Context trace_id |
logRecord.trace_id |
traceId (if available) |
|
Association Context span_id |
logRecord.span_id |
span_id (if available) |
|
Association Context <attribute_name> |
logRecord.attributes → sap.exm.related.<attribute_name> |
SAP Extension |
|
Exception Category |
logRecord.attributes → sap.exm.category |
SAP Extension (label visible in UI). |
|
Association Context correlation_id |
logRecord.attributes → sap.exm.correlation_id |
SAP Extension |
|
Collection Context aggregated exception count |
logRecord.attributes → sap.exm.aggregated_count |
SAP Extension. Optional. Number of log occurrences. |
|
CollectionContext |
logRecord.attributes |
List of key-value pairs depending on the exception category attributes |
Business Service Management Mapping
In addition to the common mapping rules, the table below describes the mapping rules specific to Business Service Management scenario.
|
BSM |
OTLP LogRecord |
Comment |
|---|---|---|
|
subject |
logRecord.body |
Event type eventphase: servicename Example: Maintenance [Announcement]: Brain Software |
|
Constant "9" |
logRecord.severityNumber |
|
|
Constant "INFO" |
logRecord.severityText |
|
|
Announcement/start/update/Closure timestamp |
logRecord.timeUnixNano |
Time when announcement/update/start/closure happened. BSM will not distinguish between Timestamp and observed timestamp. |
|
Not used |
logRecord.observed |
|
|
Not used |
logRecord.trace_id |
|
|
Not used |
logRecord.span_id |
|
|
Attributes |
||
|
eventId |
sap.bsm.event.id |
The CALM internal event id e.g. "1cea186c-52f7-477d-84e8-d6475251e566" |
|
eventName |
sap.bsm.event.name |
Event name e.g. "EV23456789" or "Regular weekend maintenance" |
|
eventSource |
sap.bsm.event.source |
Whether the event was created by SAP, by customer in SAP Cloud ALM, or from inbound API Possible values: "SAP", "Customer", "API" or "External" tbd |
|
eventphase |
sap.bsm.event.phase |
Phase of the event . Possible values: "Announcement", "Detection", "Status Update", "Start", "Completion", "Resolution", "Root Cause Analysis" |
|
eventDescription |
sap.bsm.event.description |
Description of the event formatted as http text (only if showDescription = X ) |
|
eventStartTime |
sap.bsm.event.start_time |
Start Time of Event in UTC |
|
eventEndTime |
sap.bsm.event.end_time |
End Time of Event in UTC |
|
eventType |
sap.bsm.event.domain |
Event type for which the client needs to get events. Available values: "Maintenance", "Degradation", "Disruption", "Communication" |
|
CSSIncidentID |
sap.bsm.event.cssid |
CSS id of related ticket (if exists) |
|
affectedServices |
sap.bsm.event.affected_services |
If logs are requested for a business service but events were created for services or systems of the business service, then this contains information about affected services |
Integration Monitoring Mapping
In addition to the common mapping rules, the table below describes the mapping rules specific to Integration Monitoring scenario.
Version 1
|
Integration Monitoring |
OTLP Log Record |
Comment |
|---|---|---|
|
Start time of the message |
logRecord.timeUnixNano |
|
|
json complex structure describing the message payload |
logRecord.body |
See below Caution: the body is stringified; the example below is unstringified for better readability. |
Version 2
|
Message Section |
Field Name |
LogRecord |
|---|---|---|
|
Timestamp in nano seconds |
timeUnixNano |
|
|
W3C trace ID |
traceid base 64 encoded |
|
|
Header |
Message Category |
sap.im.category |
|
Source event ID |
sap.im.source_eventid |
|
|
Event start timestamp |
sap.im.event_timestamp |
|
|
EventEndTimestamp |
sap.im.event_end_timestamp |
|
|
Direction of the message |
sap.im.direction |
|
|
Status of the message |
sap.im.status |
|
|
Status group of the message |
sap.im.status_group |
|
|
Status text of the message |
sap.im.status_text |
|
|
Passport |
Transaction ID |
sap.im.passport.transaction_id |
|
Connection ID |
sap.im.passport.connection_id |
|
|
Connection counter |
sap.im.passport.connection_counter |
|
|
Component name |
sap.im.passport.component_name |
|
|
Component type |
sap.im.passport.component_type |
|
|
Previous component |
sap.im.passport.previous_component |
|
|
Service |
sap.im.passport.service |
|
|
Action |
sap.im.passport.action |
|
|
Action type |
sap.im.passport.action_type |
|
|
Root context ID |
sap.im.passport.root_context_id |
|
|
Application data sent by the managed component as name-value pairs |
Parameter |
sap.im.application_data.<paramid> |
Monitoring Event Mapping
|
OTLP LogRecord |
IEP |
Comment |
|---|---|---|
|
body |
Short description of the event |
|
|
timeUnixNano |
timestamp converted to timeUnixNano |
Should be the timestamp of the monitoring event (When it was detected, updated etc) in UTC |
|
observedTimeUnixNano |
observed timestamp |
|
OTLP LogRecord Attributes |
IEP |
Comment |
|---|---|---|
|
sap.iep.event.event_config_id |
eventConfigId |
CALM internal event config id |
|
sap.iep.event.event_name |
eventname |
Event name |
|
sap.iep.event.event_state |
eventstate |
State of the event. Ex: CREATE, UPDATE, CLOSE |
|
sap.iep.event.event_subtype |
eventsubtype |
Metric technical name Ex: "calm.im.cpi.msg" |
|
sap.iep.event.major_update |
majorupdate |
Indicates if the event is a major update or not |
|
sap.iep.event.rating |
rating |
Rating/severity of the event Possible values: OK, INFO, WARNING, ERROR, FATAL |
|
sap.iep.event.subject |
subject |
Short description of the event |
|
sap.iep.event.tags |
tags |
Additional attributes/data from use case |
|
sap.iep.event.use_case |
usecase |
Use case that sent the event to IEP Possible values: CALM.EXM, CALM.IM, CALM.BPMON, CALM.BSM, CALM.HM, CALM.JM, CALM.SUM |
Job & Automation Monitoring Mapping
In addition to the common mapping rules, the table below describes the mapping rules specific to the Jobs & Automation Monitoring scenario.
Remark: Job and Automation Monitoring have two types of source: OpenTelemetry sources and non OpenTelemetry sources.
LogRecord for Job Definition
provider = jam.def
|
OTLP LogRecord |
JAM |
|---|---|
|
timeUnixNano |
Event timestamp |
|
observed |
Event observed timestamp |
|
Attributes |
|
|
sap.jam.job_type |
Type of Job (e.g. NW_ABAP_JOB). |
|
sap.jam.def.job_id |
ID of Job |
|
sap.jam.def.job_name |
Name of job Displayed in SAP Cloud ALM JAM list and can be used for filtering |
|
sap.jam.def.job_description |
Description of bob |
|
sap.jam.job_context |
Additional header level information, that should not be part of the name, but is needed for filtering |
|
sap.jam.schedules[] |
Array of schedule details JSON string |
|
sap.jam.def.schedules[].id |
ID of schedule |
|
sap.jam.def.schedules[].description |
Verbal description |
|
sap.jam.def.schedules[].cron |
|
|
sap.jam.def.schedules[].repeatInterval |
Repeat interval |
|
sap.jam.def.schedules[].scheduledBy |
User that scheduled the executions, displayed in JAM UI |
|
sap.jam.def.schedules[].scheduledPriority |
Priority of job execution. 1, 2, 3 (i.e. prio 1 or 2 or 3) |
|
sap.jam.def.executables[] |
Array of step execution details JSON string |
|
sap.jam.def.executables[].id |
ID of executable or step, 1 for first step, 2 for second step, but can also be another stable ID (needs to be sent with every execution as well) |
|
sap.jam.def.executables[].name |
Name of executable or step |
|
sap.jam.def.executables[].executedBy |
User that executes the job step |
|
sap.jam.def.add_info[] |
Array of definition additional info JSON string |
|
sap.jam.def.add_info[].name |
Name |
|
sap.jam.def.add_info[].value |
Value |
LogRecord for Job Execution
provider = jam.run
|
OTLP LogRecord |
JAM |
|---|---|
|
timeUnixNano |
Event timestamp |
|
observedTimeUnixNano |
Event observed timestamp |
|
Attributes |
|
|
sap.jam.run.job_id |
ID of job |
|
sap.jam.run.job_name |
Name of the job |
|
sap.jam.run.job_type |
Type of the job |
|
sap.jam.run.schedule_id |
ID of schedule |
|
sap.jam.run.id |
Run ID of job execution |
|
sap.jam.run.planned_start_time |
Planned start time of job execution DD/MM/YYYY hh:mm:ss.ms UTC |
|
sap.jam.run.start_time |
Start time of job execution DD/MM/YYYY hh:mm:ss.ms UTC |
|
sap.jam.run.end_time |
End time of job execution DD/MM/YYYY hh:mm:ss.ms UTC |
|
sap.jam.run.status_rating |
Technical Status (grey as long as job is running, green, if all job steps could be started successfully, else red or yellow) One of the following: 0, 1, 2, 3 (grey, green, yellow, red) |
|
sap.jam.run.status_text |
Job execution status text One of the following: Scheduled, Started, Finished, Failed |
|
sap.jam.run.status_text_long |
Job Execution Status long text |
|
sap.jam.run.appl_status_long_text |
Job Application Status long text |
|
sap.jam.run.appl_status_text |
Job Application Status |
|
sap.jam.run.appl_status_rating |
Job Application Status rating (grey as long as job is running, green, if all no errors or warnings with data processing successfully, else red or yellow). One of following 0, 1, 2, 3 (grey, green, yellow, red) |
|
sap.exm.correlation_id |
Exception Monitoring correlation-id. |
|
sap.jam.run.executables[] |
Not available with provider "jam.run". Use provider "jam.run_step" to retrieve this information. |
|
sap.jam.run.add_info[] |
Additional information related to run data |
|
sap.jam.run.add_info[].name |
for example for navigation URL: NAV_URL |
|
sap.jam.run.add_info[].value |
LogRecord for Job Step Execution
provider = jam.run_step
|
OTLP LogRecord |
JAM |
|---|---|
|
timeUnixNano |
Event timestamp |
|
observedTimeUnixNano |
Event observed timestamp |
|
Attributes |
|
|
sap.jam.run_exec.job_id |
ID of job |
|
sap.jam.run_exec.job_name |
Name of Job (displayed in CALM JAM list and can be used for filtered) |
|
sap.jam.run_exec.job_type |
Type of the job |
|
sap.jam.run_exec.runId |
Run ID of Job execution |
|
sap.jam.run_exec.schedule_id |
Schedule ID of job |
|
sap.jam.run_exec.id |
ID of executable or step, 1 for first step . |
|
sap.jam.run_exec.execRunId |
Run ID of Job Step (in case of chain of jobs) |
|
sap.jam.run_exec.parentId |
ID of preceding executable or step |
|
sap.jam.run_exec.context |
|
|
sap.jam.run_exec.name |
If dynamic, i.e. different from executable name that has been sent with definition |
|
sap.jam.run_exec.startTime |
DD/MM/YYYY hh:mm:ss.ms UTC |
|
sap.jam.run_exec.endTime |
DD/MM/YYYY hh:mm:ss.ms UTC |
|
sap.jam.run.exec.statusRating |
Technical Status (grey as long as step is running, green, if step finished successfully, else red or yellow). One of following: 0, 1, 2, 3 (grey, green, yellow, red) |
|
sap.jam.run_exec.statusText |
|
|
sap.jam.run_exec.applStatusRating |
Application Status (grey as long as step is running, green, if step finished without application errors or warnings, else red or yellow). One of following: 0, 1, 2, 3 (grey, green, yellow, red) |