Offline Overview¶
Even though today's modern cellular networks are reliable and fast, many times users encounter situations where apps are not working properly due to low network bandwidth or even no network connection at all. This may occur in non-urban environments or in areas where network coverage can't be guaranteed, such as basements, oil rigs, retail stores, warehouses and the like. Nevertheless, today's user expectation is that important business apps work anytime, everywhere, regardless of network availability.
An example of such an offline application is one that is used by field service personnel, who download service information and data from the back end at the beginning of a day, make changes to the offline data throughout the day, then upload the modification requests to the back end at the end of the day. This use case requires a sophisticated offline technology that can support app developers in delivering an uninterrupted user experience as part of a reliable and fast mobile app that goes far beyond traditional caching. Offline-enabled mobile apps require more thought and planning in terms of performance, data volume, concurrency and error handling than online apps. The additional effort pays off quickly by providing a more reliable and user-friendly app.
Procedure¶
- Add the Offline feature to your Mobile App configuration in the SAP mobile service cockpit.
- Integrate the logging feature of the SDKs into your application.
- Initialize the Offline store.
- Use OData CRUD calls while being offline.
- Download and upload data while connected to mobile services.
- Customize offline synchronization behavior and performance in the SAP mobile service cockpit.
Feature Scope¶
Feature | Description |
---|---|
CRUD | OData offline supports full CRUD operations while the user is offline. |
View offline configuration settings | View offline configuration settings in the SAP mobile service cockpit. |
View usage statistics for offline OData applications | Administrators can view request and response-time usage statistics for offline OData applications in the SAP mobile service cockpit. Statistics are gathered for offline data store operations such as build, refresh (download) and flush (upload). |
Offline data security | The local data storage used for offline access is encrypted on the device. When the Offline Store Upload API is implemented, users can securely upload local database files to the server. |
OData Version 2 data sources | Offline OData supports access to OData services following the Version 2 specifications, with additional Version 4 metadata annotations. |
OData Version 4 lambda operator | Offline store supports OData Version 4 lambda operators any and all. |
Conflict detection | The use of ETags allows conflict detection, enabling developers to notice data modifications on the same entity and react accordingly in their app. |
Handling of failed requests | Allows developers to write robust apps that can recover from business logic errors that happened while offline. |
Media resources | OData Offline supports handling of media resources provided by the back-end service. |
Repeatable requests | To ensure data consistency, we support the prevention of repeatable requests in Offline OData. |
Upload of local store | For root cause analysis, developers can extend their apps to upload the local data store to mobile services. |
CLI tools | Local tooling for troubleshooting offline scenarios. |
Event Log | Local-only entity set to view the log of past system offline system events. |
Progress API | Enables developers to inform users about ongoing data synchronization. |
Request Queue Optimization | Built-in heuristics that reduce the number of requests that get sent to the back end. |
Transaction Builder | Allows grouping of CUD operations into transactions (OData change sets). |
Undo changes | Undo local modifications of entities before uploading to the back end. |
Complex objects graphs | Allows the local creation of complex entity relations while being offline. |
How It Works¶
Caching vs. Offline-First¶
A common source of confusion is how our Offline technology relates to traditional caching:
- Caching
- A caching solution will cache the most-recently used (MRU) data
- A fixed maximum cache size is specified, data is evicted based on least-recently used (LRU)
- If you haven’t accessed data recently, it won’t be available when you go offline
- Assumes the length of time the user will be offline will be short
- Good as a way of improving performance
- Likely greatly restricts what operations the user is capable of doing while offline (e.g. read-only)
- Likely gives incorrect answers to aggregation queries while offline
- Best suited for creating improving the performance and increasing the robustness of online apps
- Offline-first
- The application proactively pulls down the data subset that the user may need eventually
- No fixed cache size, grows as large as demanded (which is both a pro and a con)
- All data the user will need while offline is available, no restrictions on usage
- Also good as a way of improving performance with one caveat: Potentially long initial download time, depending on how data is sliced
- Capable of accurately representing the responses the back end would give, with some caveats
- Best suited for apps which are expected to be used while offline for non-trivial periods of time
- Covers a wide range of operations on local data (e.g. create, read, update, delete (CRUD))
Underlying Technology¶
Mobile Service Offline OData is built on top of in-house technology, which has been designed and used in synchronizing
back-end data to mobile devices for over 20 years. On the client side, SAP is using UltraLite databases, a proprietary
standard specifically designed for mobile devices. On the device, there are two UltraLite databases, the store database,
which stores the data, and the request queue database, which stores the requests between client and server. The store
database stores the user’s view of the data, which includes server-provided data, client-generated data or both, on an
entity-by-entity basis. It also includes many other tables, such as those which store the relationships between those
entities. The request queue database stores the modification requests (POST
, PUT
, PATCH
, DELETE
) made by the user. It
also includes many other tables, such as those which support request optimization. This approach to database
synchronization is what makes SAP Offline OData very powerful in comparison to other vendors: Since we are able to
replay messages flowing between client and server, server-side auditing becomes possible in the first place.
Downloading Data to Clients¶
When the data is downloaded to the client device for the first time, it is done in a declarative way, based on a
supplied list of so-called "defining requests" or "defining queries", both denoting the same concept. Defining queries
are genuine OData queries which indicate what subset of the data on the back end to synchronize. Since they are built
on top of OData, these queries are very powerful: For instance, it is possible to synchronize full data sets (e.g. the
full entity set SalesOpportunities
), limit it to a subset by means of filtering (e.g.
SalesOpportunities?$filter=Region eq 'EMEA'
), and much more.
When the client first connects to mobile services, these queries get sent to the back end. Then, the initial store database is populated in mobile services and subsequently sent to the client as a preassembled binary. While this may sound surprising at first, this enables mobile services to efficiently synchronize shared data to multiple clients without having to do superfluous back-end roundtrips. Once the client store database has been populated and downloaded, any query can be issued locally. When subsequent downloads occur, the queries are again sent to the back end, this time with delta tokens, so that only modified data is sent to the client instead of full databases. In the process, any client-generated entities are discarded, the server-provided entities are updated, added or removed, then the request queue is replayed, recreating the client-generated entities.
Uploading Data From Clients¶
In mobile services Offline OData, requests are the one source of truth; the local data is merely a guess, albeit a
highly-educated one. When the user requests an upload, the Offline OData client will send those requests to mobile services, which will in turn relay those requests to the back end. In the case of error responses, a
virtualErrorArchive
entity set is populated on the client, allowing the user to ultimately see and/or resolve errors
that occurred on the back end when processing the requests. In the case of POST
requests, the response is processed to
enable the mapping between temporary, client-generated keys with the real, server-generated keys. Note that before
sending the requests, the Offline OData client will, if so configured, run one or more of several optimization
algorithms to re-order and/or consolidate requests to help reduce back-end processing time.
Delta Synchronization¶
When a download happens, the data on the client needs to be updated to match what is on the server. A naive way to do
this would be to send all the data down again. Under certain circumstances, this might be necessary, but under normal
conditions, this will be much too slow and expensive. The OData spec provides us with a better solution: Delta
queries.
The OData producer can supply information about the changes, or deltas. In the initial GET
request, a delta token
(typically a timestamp) is returned. When the subsequent GET
request is made, the delta token is passed in as a query
option. Seeing that, the back end only sends the data that has changed since the original timestamp. Calculating deltas
can range from relatively simple to extremely complex. Some OData producers don’t provide deltas, so mobile services
can provide it by means of a fallback algorithm. This is nowhere near as good, but better than not having deltas at
all.
In case of doubt, we recommend building mobile app back ends with Mobile Back-end Tools, which are included for mobile services customers. Back ends built with this technology are optimized for mobile use cases, and therefore feature, among other things, sophisticated delta support by default.
Repeatable Requests¶
To ensure that a request has successfully reached the back end, a client needs to get the response back. Sometimes,
perhaps but not limited to network issues, the response isn’t received. When this happens, the client doesn’t know
whether a) it was sent to the back end and the response was lost or b) it never made it to the back end. As a result,
the client is obligated to resend it. However, some requests, notably POST
requests, are not idempotent: Issuing them
twice can lead to duplicate records being created or primary-key-not-unique exceptions, so sending these requests to the
back end more than once can be a problem. Repeatable requests solve this as follows: The back end (or, failing that, mobile services) will store the response. If the request is received again, it resends the original response instead of
processing the request again.
Transaction Merging¶
If offline for a long time, a user may end up repeatedly affecting the same logical group of entities many times. They
might make changes to WorkOrder(101)
, then create WorkOrder(102)
, then work on WorkOrder(101)
again. This can be
inefficient on the back end. In many cases, there isn’t a one-to-one mapping between the client entity and a back-end
table. As a result, creating that mapping can be expensive, and recreating it over and over again can be wasteful. By
adding request options, the user can indicate which requests belong to which logical group. Offline OData will then
group these into as few batches and change sets as possible: Ideally only one, but in some cases, it may be more. Note
that the total number of requests is not reduced, the requests are reordered.
Request Queue Optimization¶
Users often generate hundreds or even thousands of requests on the device before uploading them. Processing them all at once can be expensive on the back end, particularly if everyone does this at the same time, such as at the end of their shift. However, many requests can be merged together:
POST
,PATCH
can be consolidated toPOST
PATCH
,PATCH
can be consolidated toPATCH
POST
,PATCH
,DELETE
can be eliminated entirely
However, this is dramatically more complicated than it sounds. As an example, here are a few of the considerations that we included in the Offline OData queue optimization:
- There are many different kinds of relationships
- You need to know when entities do and don’t exist in the request queue
- You need to respect batch and change set boundaries
- Offline OData allows users to mark some requests as non-mergeable
Schema Upgrades¶
Mobile development projects often involve multiple teams, with the OData service typically being the interface between
client and back end developers. As interfaces are often negotiated, services change, and local database schemas need to
be kept in sync with what the server provides. The first part of each download operation is to check if any changes have
been made to the schema (i.e. the OData service metadata, as provided in the $metadata
route). If changes are
detected, a new entity store is automatically sent to the client. The OData standard defines what changes can be made
to the metadata of a service without introducing a new service
root. A
new request queue database is not required and all local operations will be applied to the new entity store. Operations
in the request queue that had not been uploaded to the back end should not be affected by the schema change so long as
the rules in the OData standard are followed.
Shared Data¶
Some data is user-specific, some is global: The developer and/or administrator can choose to mark some, or all, requests as shared. The responses to shared requests will be stored in mobile services to reduce the round-trip time and back end load. The expiry time of shared data can be configured, and requests made within the expiry time will be much faster. On the other hand, requests that trigger a rebuild of the shared cache will be the same speed as they would have been had it not been marked as shared.
The risk of using shared data is stale data within the bounds of the expiry time. Therefore, the decision to mark a request as shared, and the expiry time chosen, needs to be made thoughtfully.
Error Handling¶
Errors must be considered when building an offline application. When OData requests submitted by Offline OData as part
of the upload fail, the errors are returned to the device and the entities involved in the failed request are put into
an error state. When the flush completes, the errors that occurred are added to a virtual entity set called the
ErrorArchive
. The ErrorArchive
can be accessed like any other OData entity set in the OData model. After the
flush it is up to the client to fix any errors. This is typically done by issuing an OData request to fix the problem.
When a flush is called with failed requests in the queue, a merge algorithm is run to merge the failed requests with new
requests referencing the same entity to attempt to create requests without the errors.
Note
Please note that this kind of error typically relates to business rules and the corresponding input validation: They can be avoided by smart client-side input validation and, potentially, by rethinking business processes.
Update Conflicts¶
One particular category of error that must be considered is how to handle update conflicts; i.e. when multiple clients
make changes to the same data. OData supports conflict detection through the use of HTTP ETags, a form of optimistic
locking: The simplest back ends will use a timestamp-based approach to generate the ETag. When an entity is updated, the
client specifies the timestamp (ETag) of the entity it is changing. If the timestamp does not match the current server
version of the request, is rejected. Offline OData applications can use ETags to detect and prevent conflicts from
happening: The rejected requests are added to the ErrorArchive
for the client to fix. Another common approach in
offline applications that can be considered is “last one wins”: The back end does not detect conflicts and allows
the last modification to be performed. However, this approach is not recommended, as it may result in loss of data.
Note
Please note that this kind of error typically relates to process issues. For instance, it may be a bad idea to allow multiple employees to access and edit the same data records concurrently in the first place, because this raises the question of how the change should be merged. Can the end users get together and figure out a solution? Is an additional back-office resource required to mediate? It is important to understand that this is not a technology issue, but a process issue, which requires thorough consideration when planning offline-enabled apps of any kind.
Sample Walkthrough¶
The following are fictional request walkthroughs to illustrate the inner workings of Offline OData as described above.
- During the initial download,
WorkOrder(101)
is added to the local store- In the
WorkOrder
table, a new row is added, marked as being a server row and as active
- In the
- On the device, the user issues a
PATCH
request forWorkOrder(101)
- The
PATCH
request is added to the request queue table - In the
WorkOrder
table, a new row is added, marked as being a local row and as active - In the
WorkOrder
table, the server row is marked as being inactive
- The
- On the device, the user issues a
GET
request forWorkOrder(101)
- The
GET
request is translated into aSELECT
on theWorkOrder
table for the active row - The row is transformed into an OData response and returned
- The
-
The user issues another download
- The back end notices that another user has updated
WorkOrder(101)
, and sends it - The server row in the
WorkOrder
table is updated - The local row is discarded, then the
PATCH
request is replayed, recreating it
- The back end notices that another user has updated
-
The user creates a new work order,
WorkOrder(102)
- The
POST
request is added to the request queue table - In the
WorkOrder
table, a new row is added, marked as being a local row and as active - Note that there is no server row; also note that the entity ID is temporary
- The
- On the device, the user issues an upload
- The
POST
request is sent to the back end - The response is examined, and the real entity ID is extracted from it
- A mapping between the temporary entity ID and the real entity ID is established
- The
POST
request is marked as sent
- The
- On the device, the user issues a download
- The back end sends the new entity
- A server row for
WorkOrder(102)
is added to theWorkOrder
table and marked as active - The local row is removed
- The
POST
request, having now been resolved, is removed