Working with the C++ Subengine to Create Operators

SAP Data Hub Modeler subengines enable users to code their own operators in a particular programming language and make them available for use from the SAP Data Hub Pipeline Modeler user interface.

Introduction

The C++ subengine can detect and execute operators that were compiled into shared objects ( *.so ). When a user runs a graph on the modeler, the main engine (which also serves the UI over HTTP) breaks it into large subgraphs such that every operator in the same subgraph can be run by the same subengine. The main engine then executes a subengine process for each subgraph.

When launched, the C++ subengine does the following:

Looks for and registers operators.
Initializes the mechanisms for communicating with the main engine.
Receives its subgraph from the main engine.
Instantiates the processes in the subgraph and initializes them.
Sets up the connections among processes.
Starts the graph, handles its input, invokes user-supplied handlers, and writes the output.
When instructed by the main engine to stop, or when a fatal error occurs, it cleans up and terminates.

Quick Start

#include <v2/subengine.h>

// Port handler
const char* echo_input( v2_process_t proc, v2_datum_t datum )
{
  // Write the input unchanged
  v2_process_output( proc, "output", datum );
  return NULL; // No error
}

// Shutdown handler (optional)
const char* echo_shutdown( v2_process_t proc )
{
  // Clean up echo's resources here.
  return NULL;
}

// Init handler
const char* echo_init( v2_process_t proc )
{
  // Call echo_input whenever port "input" receives data
  v2_process_set_port_handler( proc, "input", echo_input );
  // Call echo_shutdown when the process is stopped
  v2_process_set_shutdown_handler( proc, echo_shutdown );
  return NULL;
}

// Init function
// Remove `extern "C"` when compiling as pure C
extern "C" V2_EXPORT void init( v2_context_t ctx )
{
  // Create an operator called "Echo" with ID "demo.echo" and call
  // echo_init when initializing its processes.
  auto op = v2_operator_create( ctx, "demo.echo", "Echo", echo_init );
  // Add an input port called "input" of type string.
  v2_operator_add_input( op, "input", "string" );
  // Add an output port called "output" of type string.
  v2_operator_add_output( op, "output", "string" );
}