Modeling Guide for SAP Data Hub

Kafka Consumer 2

The Kafka Consumer operator works as a Kafka client that consumes records/messages from a Kafka cluster. It is compatible with Kafka versions 0.9.x and 0.10.x. It is not compatible with Kafka version 0.8.x. When using Kafka 0.8.x, use the Kafka Consumer operator, which is compatible with Kafka 0.8.x.

Configuration Parameters

Configuration Parameters

Parameter

Type

Description

maxWaitTime

int

The maximum amount of time in milliseconds that the broker waits for the consumer.

Default: 500

kafkaVersion

string

Mandatory. The version of Kafka installed.

Default: "0.9.0"

brokers

string

 Mandatory. Each node in the cluster.

Default: "localhost:9092"

topics

string

Mandatory. The category name to which records are published.

Default: "test_topic"

groupId

string

Mandatory. Holds all information for a consumer that is part of the group.

Default: "test_group"

offset

string

Identifies each record on partition.

Default: "newest"

autoCommit

bool

Specifies whether to automatically commit the message.

Default: true

maxMessageBytes

int

The maximum permitted size of a message.

Default: 1000000

useTLS

bool

Flag that enables using TLS/SSL authentication.

Default: false

insecureSkipVerify

bool

Controls whether a client verifies the server's certificate chain and host name.

Default: false

tlsCAFile

string

Certificate authority file path.

Default: ""

authenticate

bool

Flag that enables SASL authentication.

Default: false

username

string

Username if using kafka's SASL/PLAIN authentication.

Default: ""

password

string

Password if using kafka's SASL/PLAIN authentication.

Default: ""
Connection Parameters
Parameter Type Description
connectionType string "Regular" will get connection information from each property, while "connection management" will use connection property for connection information.

Default: regular
connection object Holds information about connection information for the services.
configurationType string Which type of connection information will be used: Manual (user input) or retrieved by the Connection Management Service.

Default: ""

connectionID string The ID of the connection information to retrieve from the Connection Management Service.

Default: ""
connectionProperties object All the connection properties for manual input:
brokers (type string)
Mandatory. Each node in the cluster.
Default: "localhost:9092"
topics (type string)
Mandatory. Name of topics from which messages are read and sent to pipeline (consumed). Each topic name is separated by a ,.
Default: test_topic
groupId (type string)
Mandatory. Holds all information for a consumer that is part of the group.
Default: test_group
insecureSkipVerify (type bool)
Controls whether a client verifies the server's certificate chain and host name.
Default: false
tlsCAFile (type string)
Certificate authority file path.

Default: ""

username (type string)
Username if using kafka's SASL/PLAIN authentication.

Default: ""

password (type string)
Password if using kafka's SASL/PLAIN authentication.

Default: ""

Input

Input

Type

Description

markOffset

message

Marks the provided message as processed.

Output

Output

Type

Description

message

message

The stream of records produced.