Modeling Guide

Avro Ingest Mod via Disk

From a Kafka broker, a series of messages are read, each consisting of Avro records. Each message is passed to the preingestor, which creates a message with record objects. Finally, this message is passed to the ingestor, which stores the records in the Vora disk engine and emits the commit token associated with each message.

Prerequisites

  • You need a running Vora instance and a Kafka broker.

Configure and Run the Graph

Follow the steps below to run the example from the Data Pipeline UI:
  1. In the left panel, select the Graphs tab and navigate to com/sap/demo/vora/ingestion/avro_ingestion_example_disk_go.
  2. Check the configuration of the consumer2 node: brokers and topics
  3. Check the configuration of the ingestor node: dsn
  4. In the tool bar, select Run (play button).
  5. The Status panel indicates if the graph is running.
  6. Use the context menu Open UI of the Wiretap node to open the wiretap.
  7. Send Avro messages to the specified Kafka topic.
  8. The wiretap opens and you see the commit tokens.
  9. Stop the graph and change the generator’s batchSize and run the graph again.