Published Data Lost When Node Fails

Issue: A subscriber receiving data through a stream does not receive all the data sent by the publisher before the ESP node shuts down unexpectedly.

Solution 1: Replace the Stream with a Long Retention Time Window

Replace the stream feeding the subscriber with a window configured with a retention time long enough to allow the subscriber to reconnect when the node comes back up or (if the node is in active-active HA mode) fails over to its secondary instance. For example, if it takes the subscriber a minute to reconnect, configure a 2-minute retention time.

Note The subscriber must filter out any delete operations coming from the window and view only inserts.

See the SAP Event Stream Processor: Developer Guide for information on data retention policies for windows.

Solution 2: Replace the Stream with a Guaranteed Delivery Window

Replace the stream feeding the subscriber with a window on which guaranteed delivery (GD) is enabled.

Guaranteed delivery (GD) uses log stores to ensure that a GD subscriber registered with a GD stream or window receives all the data processed by that stream or window even if the client is not connected when the data is produced. GD is supported on streams and windows and each GD stream or window requires a log store.

Note GD-enabled windows are not recommended on a node configured for active-active HA mode. The shared disk requirements for GD log stores are incompatible with the continuous synchronization that enables an active-active primary instance to fail over quickly to its secondary instance.

For a window assigned to a memory store, or a stream, consider using persistent subscribe pattern (PSP) instead of GD.

See the SAP Event Stream Processor: Developer Guide for information on guaranteed delivery and PSP.