Raw Data Outbound Logs API

Introduction

The SAP Cloud ALM Raw Data Outbound Logs API is used to read logs produced by the different SAP Cloud ALM monitoring use-cases with the OTeL format in PULL mode. This API can be used to integrate data from an open source, in-house tool or third-party tool by exporting SAP Cloud ALM data in the OTel format.

The number of log records returned by the API depends on the time period and time filter parameters used during the API calls.

The following metrics data providers are supported:

Log Providers

SAP Cloud ALM Monitoring Use Cases

exm.im

Exception Monitoring for Integration Monitoring

exm.jam

Exception Monitoring for Job Monitoring

bsm

Business Services Management (status events)

im

Integration Monitoring

iep

Monitoring Events (see prerequisites)

jam.def

Job Automation Monitoring Definition Logs

jam.run

Job Automation Monitoring Runtime Logs

jam.run_step

Job Automation Monitoring Runtime Step Logs

Data Protection and Privacy

The SAP Cloud ALM API Raw Data Outbound Logs can return personal and sensitive data which must not be accessed or stored without technical and organizational measures to safeguard data protection and privacy.

Check the following guide for more information: https://help.sap.com/docs/cloud-alm/setup-administration/data-protection-privacy?locale=en-US

Type

Pull API

URL Format

Logs data are retrieved from SAP Cloud ALM via a dedicated HTTP API end-point:
  • https://<tenant name>.<provider name>/api/calm-logs/v1/logs

Documentation

Log data are retrieved from SAP Cloud ALM via an HTTP API end-point documented in the SAP API HUB.

Operations

  • GET /logs

    • calm-logs/v1/logs GET

Authorizations

Access to the logs API require the following security scopes:

Security Scope

Description

calm-api.logs.read

Mandatory security scope to access any CALM logs

calm-api.exm.read

To access Exception Monitoring use case data

calm-api.exm.personal.read

To read log in Exception Monitoring use case.

This security scope is necessary to allow extraction of data that could contain personal and sensitive data.

calm-api.bsm.read

To get access to Business Services Management data

calm-api.integration-monitoring.read

To get access to the Integration Monitoring data

calm-api.integration-monitoring.personal.read

To read log in Integration Monitoring use case.

This security scope is necessary to allow extraction of data that could contain personal and sensitive data.

calm-api.iep.read

To get access to the Monitoring Event data

calm-api.jobs.read

To get access to the Jobs & Automation Monitoring data

calm-api.jobs.personal.read

To get access to the Jobs & Automation Monitoring personal data

A request without security scope authorisation always receives an authorisation exception (HTTP 403).

Examples

The following call demonstrates an access to the "Exception monitoring" for the last 5 minutes with a filter on service id and category:
  • https://{{CALM_URL}}/api/calm-logs/v1/logs?version=V1&period=C1H&provider=exm.im&serviceId=3c8c7cd1-a346-4af1-a52e-428e53848669&category=ABAP Job Application Log&format=protobuf-binary

The following call demonstrates an access to the "Integration Monitoring" for the last 60 minutes with a filter on service id and categories.
  • https://{{CALM_URL}}/api/calm-logs/v1/logs?provider=im&serviceId=93794776-c9a3-4b80-8532-38d408628bd9,93794776-c9a3-4b80-8532-38d408628bd9&category=SAP Integration Suite Messages,Concur Objects,PI Messages,IDoc&format=protobuf-json&period=60M&limit=100&onLimit=TRUNCATE

API Details

The SAP Cloud ALM Raw Data Outbound Logs request the following parameters:

Query parameters

Name

Description

Comments

format

Expected output format of the logs api. You can select between text, protobuf-binary or protobuf-json format.

Supported Values:
  • protobuf-binary

  • protobuf-json

String

Default: protobuf-binary

provider

The log data provider accessed though the logs api.

Supported values:
  • exm.im

  • exm.jam

  • bsm

  • im

  • iep

  • jam.def

  • jam.run

  • jam.run_step

String

Mandatory

version

Version of the logs provider.

String

Example: v1

Default: Oldest non-deprecated version.

period

Semantic period to collect. This parameter can be used instead of parameters “from” and “to”; it is converted to “from” and “to”. Parameter “to” is set to Now, where 'Now' is the current UTC unix time on SAP Cloud ALM side, and parameter “from” is calculated by subtracting “period” to parameter “to”. If parameters “to” and/or “from” are provided in addition to “period”, then an error is returned (incompatible parameters “from”/”to” and “period”). Possible values: <X>M where X is in [1,60] unless provider authorizes larger from/to period (e.g. bsm has no restriction for requested from/to period).​.​
  • Examples

    • 5M

String

Default: 5M

from

UTC timestamp indicating the beginning of the collection period. The largest possible period to collect is 1 hour; if the period (from/to) goes above one hour, then an error is returned. Caution: “from” timestamp is inclusive.

Optional

Format: YYYYMMDDHHmmss

to

UTC timestamp indicating the end of the collection period. The oldest timestamp accepted for parameter “to” is defined by the Logs Provider, if “to” goes beyond this limit an error should be returned by the Logs Provider. If “to” is not provided or if “to” value is in the future, then the newest log records until Now are returned. Caution: “to” is exclusive.

Optional

Format: YYYYMMDDHHmmss

Default: Now

observedTimestamp

Boolean. Indicates if the "from" & "to" timestamps correspond to measured timestamps (observedTimestamp = false) or observed timestamps (observedTimestamp = true). Since a log measured at time t1 might be observed later at t1+delta (delta time is due to data collection infrastructure), it may be interesting to rely on observed timestamps which guarantee the presence of the logs.

Not relevant for providers bsm since they doesn't distinguish between timestamp and observed timestamp.

For provider jam.run, observedTimestamp=true is the default and the only mode supported.

For provider jam.def, period (from/to) is not used, all job definitions are always returned.

Default: false

<filters>

Log Records filters are passed as URL parameter.

The list of filters depends on the Logs Data Provider. See next chapter for a description of filters per logs data providers.

Optional

limit

Pagination support. Maximum number of log records to return. Minimum value is 1, maximum value is 1000.

Default: 100

offset

Pagination support. Number of log records to skip. If the offset goes beyond the total number of log records available, then no log records are returned (no error).

Default: 0

onLimit

Indicates what to do if the number of records goes above the maximum limit (parameter “limit”). Possible values are: ERROR (return an error), TRUNCATE: truncate the response.

Default: ERROR

Filters

Exception Monitoring

The table below describes the filters available for Exception Monitoring.

Filter Name

Description

Comment

serviceId

Comma-separated list of LMS service Ids. Minimum 1 service id, maximum 30 service ids.

Mandatory

category

Exception source category

Optional

status

Resolved, Warning or Error

Optional

message

Wildcard '*' can be used to support following scenarios:
  • Starts with: text*

  • Ends with: *text

  • Contains: *text*

  • Exact match: text

Optional

Business Service Management

Remark: only the log corresponding to the last event status is returned and not the complete action logs. For example, if an event has been announced, updated and completed today and we request the BSM logs for today, we get only the log "event completed".

The table below describes the filters available for Business Service Management.

Filter Name

Description

Comment

serviceId

LMS service Id

Mandatory. Can be for business service, cloud service or system

from

Timestamp indicating the beginning of the collection period.

from and to indicate the time when the event was updated and not necessarily the time when the event starts or stops For example:
  • If an event was announced in the past for the future the API returns a corresponding log message if from< announcement time < to

  • If an event was already started the API returns a corresponding log message if from< start time < to

BSM OTEL API does not distinguish between observedTimestamp and measured timestamp

There is no restriction on period length and on how far we can go in the past. Only restriction is from <= to.

to

Timestamp indicating the end of the collection period.

from and to indicate the time when the event was updated and not necessarily the time when the event starts or stops For example:
  • If an event was announced in the past for the future the API returns a corresponding log message if from< announcement time < to

  • If an event was already started the API returns a corresponding log message if from< start time < to

BSM OTEL API does not distinguish between observedTimestamp and measured timestamp

There is no restriction on period length and on how far we can go in the past. Only restriction is from <= to.

eventDomain

Event type for which the client needs to get events. Available values: Maintenance, Degradation, Disruption, Communication

Optional

eventPhase

Phase of the event . Available values: Announcement, Detection, Status Update, Start, Completion, Resolution, Root Cause Analysis.

Optional

showDescription

Whether the description should be returned or not . Possible values X or <space>. Default <space>

 

eventSource

source of the event. possible values: SAP, Customer, API or External (in the future )

 

Integration Monitoring

The table below describes the filters available for Integration Monitoring.

Filter Name

Description

Comment

serviceId

Comma-separated list of LMS service Ids

Mandatory

from / to

Timestamp indicating the beginning and the end of the collection period. All messages with start time between from and to timestamps will be returned by the interface.

Mandatory

from/to or period

category

Comma-separated list of Category labels as displayed in Integration Monitoring UI

Mandatory

Event Monitoring

The table below describes the filters available for Event Monitoring.

Filter Name

Description

Comment

serviceId

LMS service Id

Mandatory

from / to

Timestamp indicating the beginning and the end of the collection period. All messages with start time between from and to timestamps will be returned by the interface.

Mandatory

usecase

Monitoring usecase

Allowed values: "CALM.EXM", "CALM.IM", "CALM.BPMON", "CALM.BSM", "CALM.HM", "CALM.JM", "CALM.SUM"

Optional

Only one value

eventsubtype

Metric technical name Ex: "calm.im.cpi.msg"

Optional

Only one value

Jobs & Automation Monitoring

The table below describes the filters available for Jobs & Automation Monitoring (providers: jam.def and jam.run)

Filter Name

Description

Comment

serviceId

Comma-separated list of LMS service Ids. Minimum 1 service id, maximum 27 service ids.

Mandatory

from / to

Timestamp indicating the beginning and the end of the collection period. All messages with start time between from and to timestamps will be returned by the interface.

Mandatory

from / to or period

jobType

Job type

CF_DM_PROCESS → SAP DM Production Process

HCP_IPS_JOB → SAP Cloud Identity Provisioning Job

IRPA → SAP Intelligent RPA Job

NW_ABAP_JOB → SAP ABAP Job

NW_ABAP_PC → SAP BW Process Chain

NW_ABAP_WF → SAP Business Workflow

NW_APPL_JOB → SAP Application Job

SPA_AUTOMATION → SAP Build Automation Job

SPA_PROCESS → SAP Build Process

Optional

jobName

Job name

Optional

jobContext

Job context

Optional

name

Step name

Optional. Relevant with provider jam.run_step

Status Codes

The following status codes apply to this HTTP action:

Status Code

Description

202

Data accepted

400

Structure of the request is invalid.

Unknown parameter “XYZ”

Parameters “from”/”to” and “period” are incompatible.

Parameter XYZ has an unsupported value “XYZ”

403

Authentication failure. Security scope is missing

404

The request path is incorrect.

Unknown use-case "XYZ"

Unknown use-case version "X"

Service "XYZ" not found

408

The request took too long to reach the endpoint.

413

The payload was too big.

428

Pre-condition not full filled.

429

The request rate quota has been exceeded.

500

Internal Server Error

502

Bad Gateway

503

Service unavailable

504

Gateway Timeout

Response headers

Header

Description

x-total-count

Total number of log records. Returned only if parameter "onLimit" = "TRUNCATE"

Response Body Example

EXM - Response Body Example

BSM - Response Body Example

 

Monitoring Event - Response body example

 

Limitations

The following rules apply when accessing the SAP Cloud ALM Logs API.
  • Throughput:

    • Maximum 100 requests per minute

  • Limits:

    • Maximum log records per response: 1000 (HTTP Error Code 403 + Error Message)

Prerequisites

Before to access the Raw Data Outbound Logs API for Monitoring Event, you must enable the action "Store Event Payload for 24 Hours" on the required events. More details are available in Intelligent Event Processing documentation here.

Restrictions

The json protobuf encoding does not support the deviations described https://opentelemetry.io/docs/specs/otlp/#json-protobuf-encodingInformation published on non-SAP site; to manage correlation id use the protobuf binary format. With protobuf json format, spanId and traceId are base-64-encoded and not case-insensitive-hex-encoded-strings.

Models mapping

This section details how the SAP Cloud ALM data model is mapped to the OTLP data model.
  • Some mapping rules are common to all scenarios (see section Common mapping).

  • Some mapping rules are specific to each scenario (see dedicated scenario mapping tables).

Common Mapping

SAP Cloud ALM (source)

OTLP (target)

Comments

Service type

resourceLogs / resource / attribute / service.namespace

 

Service name

resourceLogs / resource / attribute / service.name

 

Service description

resourceLogs / resource / attribute / sap.service.display_name

 

Service attributes

list of key value-pairs resources attributes

Depends on data source

Logs data provider's version

resourceLogs / scopeLogs / scope / version

Depends on data source

Logs data provider's technical name

resourceLogs / scopeLogs / scope / name

Depends on data source

  • externalId for Cloud Service

  • systemNumber for Technical System

resourceLogs / resource / attribute / service.instance.id

 
 

resourceLogs / resource / attribute / telemetry.sdk.name

Depends on data source

 

resourceLogs / resource / attribute / telemetry.sdk.language

Depends on data source

 

resourceLogs / resource / attribute / telemetry.sdk.version

Depends on data source

Exception Monitoring Mapping

In addition to the common mapping rules, the table below describes the mapping rules specific to Exception Monitoring scenario.

LogRecord

Exception Monitoring

OTLP LogRecord

Comment

Message

logRecord.body

 

Status

logRecord.severityNumber / logRecord.severityText

 

Timestamp

logRecord.timeUnixNano

 

Saved_timestamp

logRecord.observedTimeUnixNano

 

Association Context trace_id

logRecord.trace_id

traceId (if available)

Association Context span_id

logRecord.span_id

span_id (if available)

Association Context <attribute_name>

logRecord.attributes → sap.exm.related.<attribute_name>

SAP Extension

Exception Category

logRecord.attributes → sap.exm.category

SAP Extension (label visible in UI).

Association Context correlation_id

logRecord.attributes → sap.exm.correlation_id

SAP Extension

Collection Context aggregated exception count

logRecord.attributes → sap.exm.aggregated_count

SAP Extension. Optional. Number of log occurrences.

CollectionContext

logRecord.attributes

List of key-value pairs depending on the exception category attributes

Business Service Management Mapping

In addition to the common mapping rules, the table below describes the mapping rules specific to Business Service Management scenario.

BSM

OTLP LogRecord

Comment

subject

logRecord.body

Event type eventphase: servicename

Example: Maintenance [Announcement]: Brain Software

Constant "9"

logRecord.severityNumber

 

Constant "INFO"

logRecord.severityText

 

Announcement/start/update/Closure timestamp

logRecord.timeUnixNano

Time when announcement/update/start/closure happened. BSM will not distinguish between Timestamp and observed timestamp.

Not used

logRecord.observed

 

Not used

logRecord.trace_id

 

Not used

logRecord.span_id

 

Attributes

eventId

sap.bsm.event.id

The CALM internal event id e.g. "1cea186c-52f7-477d-84e8-d6475251e566"

eventName

sap.bsm.event.name

Event name e.g. "EV23456789" or "Regular weekend maintenance"

eventSource

sap.bsm.event.source

Whether the event was created by SAP, by customer in SAP Cloud ALM, or from inbound API

Possible values: "SAP", "Customer", "API" or "External" tbd

eventphase

sap.bsm.event.phase

Phase of the event . Possible values: "Announcement", "Detection", "Status Update", "Start", "Completion", "Resolution", "Root Cause Analysis"

eventDescription

sap.bsm.event.description

Description of the event formatted as http text (only if showDescription = X )

eventStartTime

sap.bsm.event.start_time

Start Time of Event in UTC

eventEndTime

sap.bsm.event.end_time

End Time of Event in UTC

eventType

sap.bsm.event.domain

Event type for which the client needs to get events. Available values: "Maintenance", "Degradation", "Disruption", "Communication"

CSSIncidentID

sap.bsm.event.cssid

CSS id of related ticket (if exists)

affectedServices

sap.bsm.event.affected_services

If logs are requested for a business service but events were created for services or systems of the business service, then this contains information about affected services

Integration Monitoring Mapping

In addition to the common mapping rules, the table below describes the mapping rules specific to Integration Monitoring scenario.

Version 1

Log Record

Integration Monitoring

OTLP Log Record

Comment

Start time of the message

logRecord.timeUnixNano

 

json complex structure describing the message payload

logRecord.body

See below

Caution: the body is stringified; the example below is unstringified for better readability.

Version 2

LogRecord

Message Section

Field Name

LogRecord

 

Timestamp in nano seconds

timeUnixNano

 

W3C trace ID

traceid base 64 encoded

Header

Message Category

sap.im.category

 

Source event ID

sap.im.source_eventid

 

Event start timestamp

sap.im.event_timestamp

 

EventEndTimestamp

sap.im.event_end_timestamp

 

Direction of the message

sap.im.direction

 

Status of the message

sap.im.status

 

Status group of the message

sap.im.status_group

 

Status text of the message

sap.im.status_text

Passport

Transaction ID

sap.im.passport.transaction_id

 

Connection ID

sap.im.passport.connection_id

 

Connection counter

sap.im.passport.connection_counter

 

Component name

sap.im.passport.component_name

 

Component type

sap.im.passport.component_type

 

Previous component

sap.im.passport.previous_component

 

Service

sap.im.passport.service

 

Action

sap.im.passport.action

 

Action type

sap.im.passport.action_type

 

Root context ID

sap.im.passport.root_context_id

Application data sent by the managed component as name-value pairs

Parameter

sap.im.application_data.<paramid>

Monitoring Event Mapping

In addition to the common mapping rules, the table below describes the mapping rules specific to Monitoring Event scenario.

OTLP LogRecord

IEP

Comment

body

Short description of the event

timeUnixNano

timestamp converted to timeUnixNano

Should be the timestamp of the monitoring event (When it was detected, updated etc) in UTC

observedTimeUnixNano

observed timestamp

 

OTLP LogRecord Attributes

IEP

Comment

sap.iep.event.event_config_id

eventConfigId

CALM internal event config id

sap.iep.event.event_name

eventname

Event name

sap.iep.event.event_state

eventstate

State of the event. Ex: CREATE, UPDATE, CLOSE

sap.iep.event.event_subtype

eventsubtype

Metric technical name Ex: "calm.im.cpi.msg"

sap.iep.event.major_update

majorupdate

Indicates if the event is a major update or not

sap.iep.event.rating

rating

Rating/severity of the event

Possible values: OK, INFO, WARNING, ERROR, FATAL

sap.iep.event.subject

subject

Short description of the event

sap.iep.event.tags

tags

Additional attributes/data from use case

sap.iep.event.use_case

usecase

Use case that sent the event to IEP

Possible values: CALM.EXM, CALM.IM, CALM.BPMON, CALM.BSM, CALM.HM, CALM.JM, CALM.SUM

Job & Automation Monitoring Mapping

In addition to the common mapping rules, the table below describes the mapping rules specific to the Jobs & Automation Monitoring scenario.

Remark: Job and Automation Monitoring have two types of source: OpenTelemetry sources and non OpenTelemetry sources.

LogRecord for Job Definition

provider = jam.def

OTLP LogRecord

JAM

timeUnixNano

Event timestamp

observed

Event observed timestamp

Attributes

sap.jam.job_type

Type of Job (e.g. NW_ABAP_JOB).

sap.jam.def.job_id

ID of Job

sap.jam.def.job_name

Name of job

Displayed in SAP Cloud ALM JAM list and can be used for filtering

sap.jam.def.job_description

Description of bob

sap.jam.job_context

Additional header level information, that should not be part of the name, but is needed for filtering

sap.jam.schedules[]

Array of schedule details

JSON string

sap.jam.def.schedules[].id

ID of schedule

sap.jam.def.schedules[].description

Verbal description

sap.jam.def.schedules[].cron

sap.jam.def.schedules[].repeatInterval

Repeat interval

sap.jam.def.schedules[].scheduledBy

User that scheduled the executions, displayed in JAM UI

sap.jam.def.schedules[].scheduledPriority

Priority of job execution. 1, 2, 3 (i.e. prio 1 or 2 or 3)

sap.jam.def.executables[]

Array of step execution details

JSON string

sap.jam.def.executables[].id

ID of executable or step, 1 for first step, 2 for second step, but can also be another stable ID (needs to be sent with every execution as well)

sap.jam.def.executables[].name

Name of executable or step

sap.jam.def.executables[].executedBy

User that executes the job step

sap.jam.def.add_info[]

Array of definition additional info

JSON string

sap.jam.def.add_info[].name

Name

sap.jam.def.add_info[].value

Value

LogRecord for Job Execution

provider = jam.run

OTLP LogRecord

JAM

timeUnixNano

Event timestamp

observedTimeUnixNano

Event observed timestamp

Attributes

sap.jam.run.job_id

ID of job

sap.jam.run.job_name

Name of the job

sap.jam.run.job_type

Type of the job

sap.jam.run.schedule_id

ID of schedule

sap.jam.run.id

Run ID of job execution

sap.jam.run.planned_start_time

Planned start time of job execution

DD/MM/YYYY hh:mm:ss.ms UTC

sap.jam.run.start_time

Start time of job execution

DD/MM/YYYY hh:mm:ss.ms UTC

sap.jam.run.end_time

End time of job execution

DD/MM/YYYY hh:mm:ss.ms UTC

sap.jam.run.status_rating

Technical Status (grey as long as job is running, green, if all job steps could be started successfully, else red or yellow)

One of the following: 0, 1, 2, 3 (grey, green, yellow, red)

sap.jam.run.status_text

Job execution status text

One of the following: Scheduled, Started, Finished, Failed

sap.jam.run.status_text_long

Job Execution Status long text

sap.jam.run.appl_status_long_text

Job Application Status long text

sap.jam.run.appl_status_text

Job Application Status

sap.jam.run.appl_status_rating

Job Application Status rating (grey as long as job is running, green, if all no errors or warnings with data processing successfully, else red or yellow).

One of following 0, 1, 2, 3 (grey, green, yellow, red)

sap.exm.correlation_id

Exception Monitoring correlation-id.

sap.jam.run.executables[]

Not available with provider "jam.run". Use provider "jam.run_step" to retrieve this information.

sap.jam.run.add_info[]

Additional information related to run data

sap.jam.run.add_info[].name

for example for navigation URL: NAV_URL

sap.jam.run.add_info[].value

LogRecord for Job Step Execution

provider = jam.run_step

OTLP LogRecord

JAM

timeUnixNano

Event timestamp

observedTimeUnixNano

Event observed timestamp

Attributes

sap.jam.run_exec.job_id

ID of job

sap.jam.run_exec.job_name

Name of Job (displayed in CALM JAM list and can be used for filtered)

sap.jam.run_exec.job_type

Type of the job

sap.jam.run_exec.runId

Run ID of Job execution

sap.jam.run_exec.schedule_id

Schedule ID of job

sap.jam.run_exec.id

ID of executable or step, 1 for first step .

sap.jam.run_exec.execRunId

Run ID of Job Step (in case of chain of jobs)

sap.jam.run_exec.parentId

ID of preceding executable or step

sap.jam.run_exec.context

 

sap.jam.run_exec.name

If dynamic, i.e. different from executable name that has been sent with definition

sap.jam.run_exec.startTime

DD/MM/YYYY hh:mm:ss.ms UTC

sap.jam.run_exec.endTime

DD/MM/YYYY hh:mm:ss.ms UTC

sap.jam.run.exec.statusRating

Technical Status (grey as long as step is running, green, if step finished successfully, else red or yellow). One of following: 0, 1, 2, 3 (grey, green, yellow, red)

sap.jam.run_exec.statusText

 

sap.jam.run_exec.applStatusRating

Application Status (grey as long as step is running, green, if step finished without application errors or warnings, else red or yellow). One of following: 0, 1, 2, 3 (grey, green, yellow, red)