Skip to content

Offline Overview

Even though today's modern cellular networks are reliable and fast, many times users encounter situations where apps are not working properly due to low network bandwidth or even no network connection at all. This may occur in non-urban environments or in areas where network coverage can't be guaranteed, such as basements, oil rigs, retail stores, warehouses and the like. Nevertheless, today's user expectation is that important business apps work anytime, everywhere, regardless of network availability.

An example of such an offline application is one that is used by field service personnel, who download service information and data from the back end at the beginning of a day, make changes to the offline data throughout the day, then upload the modification requests to the back end at the end of the day. This use case requires a sophisticated offline technology that can support app developers in delivering an uninterrupted user experience as part of a reliable and fast mobile app that goes far beyond traditional caching. Offline-enabled mobile apps require more thought and planning in terms of performance, data volume, concurrency and error handling than online apps. The additional effort pays off quickly by providing a more reliable and user-friendly app.

Procedure

  1. Add the Offline feature to your Mobile App configuration in the SAP mobile service cockpit.
  2. Integrate the logging feature of the SDKs into your application.
  3. Initialize the Offline store.
  4. Use OData CRUD calls while being offline.
  5. Download and upload data while connected to mobile services.
  6. Customize offline synchronization behavior and performance in the SAP mobile service cockpit.

Feature Scope

Feature Description
CRUD OData offline supports full CRUD operations while the user is offline.
View offline configuration settings View offline configuration settings in the SAP mobile service cockpit.
View usage statistics for offline OData applications Administrators can view request and response-time usage statistics for offline OData applications in the SAP mobile service cockpit. Statistics are gathered for offline data store operations such as build, refresh (download) and flush (upload).
Offline data security The local data storage used for offline access is encrypted on the device. When the Offline Store Upload API is implemented, users can securely upload local database files to the server.
OData Version 2 data sources Offline OData supports access to OData services following the Version 2 specifications, with additional Version 4 metadata annotations.
OData Version 4 lambda operator Offline store supports OData Version 4 lambda operators any and all.
Conflict detection The use of ETags allows conflict detection, enabling developers to notice data modifications on the same entity and react accordingly in their app.
Handling of failed requests Allows developers to write robust apps that can recover from business logic errors that happened while offline.
Media resources OData Offline supports handling of media resources provided by the back-end service.
Repeatable requests To ensure data consistency, we support the prevention of repeatable requests in Offline OData.
Upload of local store For root cause analysis, developers can extend their apps to upload the local data store to mobile services.
CLI tools Local tooling for troubleshooting offline scenarios.
Event Log Local-only entity set to view the log of past system offline system events.
Progress API Enables developers to inform users about ongoing data synchronization.
Request Queue Optimization Built-in heuristics that reduce the number of requests that get sent to the back end.
Transaction Builder Allows grouping of CUD operations into transactions (OData change sets).
Undo changes Undo local modifications of entities before uploading to the back end.
Complex objects graphs Allows the local creation of complex entity relations while being offline.

How It Works

Caching vs. Offline-First

A common source of confusion is how our Offline technology relates to traditional caching:

  • Caching
    • A caching solution will cache the most-recently used (MRU) data
    • A fixed maximum cache size is specified, data is evicted based on least-recently used (LRU)​
    • If you haven’t accessed data recently, it won’t be available when you go offline​
    • Assumes the length of time the user will be offline will be short​
    • Good as a way of improving performance​
    • Likely greatly restricts what operations the user is capable of doing while offline​ (e.g. read-only)
    • Likely gives incorrect answers to aggregation queries while offline​
    • Best suited for creating improving the performance and increasing the robustness of online apps
  • Offline-first
    • The application proactively pulls down the data subset that the user may need eventually​
    • No fixed cache size, grows as large as demanded (which is both a pro and a con)​
    • All data the user will need while offline is available, no restrictions on usage​
    • Also good as a way of improving performance with one caveat: Potentially long initial download time​, depending on how data is sliced
    • Capable of accurately representing the responses the back end would give, with some caveats​
    • Best suited for apps which are expected to be used while offline for non-trivial periods of time​
    • Covers a wide range of operations on local data (e.g. create, read, update, delete (CRUD))

Underlying Technology

Mobile Service Offline OData is built on top of in-house technology, which has been designed and used in synchronizing back-end data to mobile devices for over 20 years​. On the client side, SAP is using UltraLite databases, a proprietary standard specifically designed for mobile devices. On the device, there are two UltraLite databases, the store database, which stores the data, and the request queue database, which stores the requests​ between client and server. The store database stores the user’s view of the data​, which includes server-provided data, client-generated data or both, on an entity-by-entity basis. It also includes many other tables, such as those which store the relationships between those entities.​ The request queue database stores the modification requests (POST, PUT, PATCH, DELETE) made by the user​. It also includes many other tables, such as those which support request optimization.​ This approach to database synchronization is what makes SAP Offline OData very powerful in comparison to other vendors: Since we are able to replay messages flowing between client and server, server-side auditing becomes possible in the first place.

Downloading Data to Clients

When the data is downloaded to the client device for the first time, it is done in a declarative way, based on a supplied list of so-called "defining requests​" or "defining queries", both denoting the same concept. Defining queries are genuine OData queries which indicate what subset of the data on the back end to synchronize​. Since they are built on top of OData, these queries are very powerful: For instance, it is possible to synchronize full data sets (e.g. the full entity set SalesOpportunities), limit it to a subset by means of filtering (e.g. SalesOpportunities?$filter=Region eq 'EMEA'), and much more.

When the client first connects to mobile services, these queries get sent to the back end. Then, the initial store database is populated in mobile services and subsequently sent to the client​ as a preassembled binary. While this may sound surprising at first, this enables mobile services to efficiently synchronize shared data to multiple clients without having to do superfluous back-end roundtrips. Once the client store database has been populated and downloaded, any query can be issued locally. When subsequent downloads occur, the queries are again sent to the back end, this time with delta tokens, so that only modified data is sent to the client​ instead of full databases. In the process, any client-generated entities are discarded, the server-provided entities are updated, added or removed, then the request queue is replayed, recreating the client-generated entities​.

Uploading Data From Clients

In mobile services Offline OData, requests are the one source of truth; the local data is merely a guess, albeit a highly-educated one. When the user requests an upload, the Offline OData client will send those requests to mobile services, which will in turn relay those requests to the back end​. In the case of error responses, a virtualErrorArchive entity set is populated on the client, allowing the user to ultimately see and/or resolve errors that occurred on the back end when processing the requests​. In the case of POST requests, the response is processed to enable the mapping between temporary, client-generated keys with the real, server-generated keys​. Note that before sending the requests, the Offline OData client will, if so configured, run one or more of several optimization algorithms to re-order and/or consolidate requests to help reduce back-end processing time.​

Delta Synchronization

When a download happens, the data on the client needs to be updated to match what is on the server​. A naive way to do this would be to send all the data down again. Under certain circumstances, this might be necessary, but under normal conditions, this will be much too slow and expensive​. The OData spec provides us with a better solution: Delta queries. The OData producer can supply information about the changes, or deltas. In the initial GET request, a delta token (typically a timestamp) is returned. When the subsequent GET request is made, the delta token is passed in as a query option. Seeing that, the back end only sends the data that has changed since the original timestamp.​ Calculating deltas can range from relatively simple to extremely complex​. Some OData producers don’t provide deltas, so mobile services can provide it​ by means of a fallback algorithm. This is nowhere near as good, but better than not having deltas at all.​

In case of doubt, we recommend building mobile app back ends with Mobile Back-end Tools, which are included for mobile services customers. Back ends built with this technology are optimized for mobile use cases, and therefore feature, among other things, sophisticated delta support by default.

Repeatable Requests

To ensure that a request has successfully reached the back end, a client needs to get the response back​. Sometimes, perhaps but not limited to network issues, the response isn’t received​. When this happens, the client doesn’t know whether a) it was sent to the back end and the response was lost or b) it never made it to the back end​. As a result, the client is obligated to resend it​. However, some requests, notably POST requests, are not idempotent​: Issuing them twice can lead to duplicate records being created or primary-key-not-unique exceptions, so sending these requests to the back end more than once can be a problem​. Repeatable requests solve this as follows: The back end (or, failing that, mobile services) will store the response. If the request is received again, it resends the original response instead of processing the request again.

Transaction Merging

If offline for a long time, a user may end up repeatedly affecting the same logical group of entities many times​. They might make changes to WorkOrder(101), then create WorkOrder(102), then work on WorkOrder(101) again​. This can be inefficient on the back end​. In many cases, there isn’t a one-to-one mapping between the client entity and a back-end table. As a result, creating that mapping can be expensive, and recreating it over and over again can be wasteful​. By adding request options, the user can indicate which requests belong to which logical group​. Offline OData will then group these into as few batches and change sets as possible​: Ideally only one, but in some cases, it may be more​. Note that the total number of requests is not reduced, the requests are reordered​.

Request Queue Optimization

Users often generate hundreds or even thousands of requests on the device before uploading them​. Processing them all at once can be expensive on the back end​, particularly if everyone does this at the same time, such as at the end of their shift. However, many requests can be merged together​:

  • POST, PATCH can be consolidated to POST​
  • PATCH, PATCH can be consolidated to PATCH​
  • POST, PATCH, DELETE can be eliminated entirely​

However, this is dramatically more complicated than it sounds. As an example, here are a few of the considerations that we included in the Offline OData queue optimization:

  • There are many different kinds of relationships
  • You need to know when entities do and don’t exist in the request queue
  • You need to respect batch and change set boundaries
  • Offline OData allows users to mark some requests as non-mergeable

Schema Upgrades

Mobile development projects often involve multiple teams, with the OData service typically being the interface between client and back end developers. As interfaces are often negotiated, services change, and local database schemas need to be kept in sync with what the server provides. The first part of each download operation is to check if any changes have been made to the schema (i.e. the OData service metadata, as provided in the $metadata route). If changes are detected, a new entity store is automatically sent to the client.​ The OData standard defines what changes can be made to the metadata of a service without introducing a new service root.​ A new request queue database is not required and all local operations will be applied to the new entity store.​ Operations in the request queue that had not been uploaded to the back end should not be affected by the schema change so long as the rules in the OData standard are followed.

Shared Data

Some data is user-specific, some is global​: The developer and/or administrator can choose to mark some, or all, requests as shared. The responses to shared requests will be stored in mobile services​ to reduce the round-trip time and back end load. The expiry time of shared data can be configured​, and requests made within the expiry time will be much faster​. On the other hand, requests that trigger a rebuild of the shared cache will be the same speed as they would have been had it not been marked as shared.​

The risk of using shared data is stale data​ within the bounds of the expiry time. Therefore, the decision to mark a request as shared, and the expiry time chosen, needs to be made thoughtfully​.

Error Handling

Errors must be considered when building an offline application​. When OData requests submitted by Offline OData as part of the upload fail, the errors are returned to the device and the entities involved in the failed request are put into an error state.​ When the flush completes, the errors that occurred are added to a virtual entity set called the ErrorArchive. The ErrorArchive can be accessed like any other OData entity set in the OData model.​ After the flush it is up to the client to fix any errors. This is typically done by issuing an OData request to fix the problem.​ When a flush is called with failed requests in the queue, a merge algorithm is run to merge the failed requests with new requests referencing the same entity to attempt to create requests without the errors.​

Note

Please note that this kind of error typically relates to business rules and the corresponding input validation: They can be avoided by smart client-side input validation and, potentially, by rethinking business processes.

Update Conflicts

One particular category of error that must be considered is how to handle update conflicts; i.e. when multiple clients make changes to the same data. ​OData supports conflict detection through the use of HTTP ETags, a form of optimistic locking: The simplest back ends will use a timestamp-based approach to generate the ETag. When an entity is updated, the client specifies the timestamp (ETag) of the entity it is changing. If the timestamp does not match the current server version of the request, is rejected.​ Offline OData applications can use ETags to detect and prevent conflicts from happening: The rejected requests are added to the ErrorArchive for the client to fix.​ Another common approach in offline applications that can be considered is “last one wins”: The back end does not detect conflicts and allows the last modification to be performed.​ However, this approach is not recommended, as it may result in loss of data.

Note

Please note that this kind of error typically relates to process issues. For instance, it may be a bad idea to allow multiple employees to access and edit the same data records concurrently in the first place, because this raises the question of how the change should be merged. Can the end users get together and figure out a solution? Is an additional back-office resource required to mediate? It is important to understand that this is not a technology issue, but a process issue, which requires thorough consideration when planning offline-enabled apps of any kind.

Sample Walkthrough

The following are fictional request walkthroughs to illustrate the inner workings of Offline OData as described above.

  1. During the initial download, WorkOrder(101) is added to the local store​
    • In the WorkOrder table, a new row is added, marked as being a server row and as active
  2. On the device, the user issues a PATCH request for WorkOrder(101)​
    • The PATCH request is added to the request queue table​
    • In the WorkOrder table, a new row is added, marked as being a local row and as active​
    • In the WorkOrder table, the server row is marked as being inactive​
  3. On the device, the user issues a GET request for WorkOrder(101)​
    • The GET request is translated into a SELECT on the WorkOrder table for the active row​
    • The row is transformed into an OData response and returned​

  4. The user issues another download​

    • The back end notices that another user has updated WorkOrder(101), and sends it​
    • The server row in the WorkOrder table is updated​
    • The local row is discarded, then the PATCH request is replayed, recreating it​

  5. The user creates a new work order, WorkOrder(102)​

    • The POST request is added to the request queue table​
    • In the WorkOrder table, a new row is added, marked as being a local row and as active​
    • Note that there is no server row; also note that the entity ID is temporary​
  6. On the device, the user issues an upload​
    • The POST request is sent to the back end​
    • The response is examined, and the real entity ID is extracted from it​
    • A mapping between the temporary entity ID and the real entity ID is established​
    • The POST request is marked as sent​
  7. On the device, the user issues a download​
    • The back end sends the new entity​
    • A server row for WorkOrder(102) is added to the WorkOrder table and marked as active​
    • The local row is removed​
    • The POST request, having now been resolved, is removed​
Last update: January 25, 2022