Storage

The data generator produces arbitrary data.

The data is written to files by Write File. The files are stored in the directory ./tmp/demo/ (where ./ is the current working directory, i.e. where the Data Pipeline Engine was started). The file names follow the scheme file_<counter>.txt.

The Move Message generates a message with the received filepath as the source, moving the file to the archived/ directory, in the same path (destination is ${dirname}/archived/${basename}).

The file content is read from the archived directory by Read File. After reading, the content is written to a terminal and the file is removed.

Configure and Run the Graph

Follow the steps below to run the example from the Data Pipeline UI:

In the left panel, select the Graphs tab and navigate to com/sap/demo/storage.
In the tool bar, select Run (play button).
The Status panel indicates if the graph is running.
Use the context menu Open UI of the Terminal node to open the terminal.
The terminal opens and you see the produced files and their content.

Optional Step: Check the files on S3

Browse the directory ./tmp/demo/ and ./tmp/demo/archived/, and check if the files are being removed.
Disconnect the input from the Remove File node and run the graph again.
Browse the directory ./tmp/demo/archived/ and check if the moved files are still there.