Modeling Guide

Kafka Consumer 2

The Kafka Consumer operator works as a Kafka client that consumes records/messages from a Kafka cluster. It is compatible with Kafka versions 0.9.x and 0.10.x. It is not compatible with Kafka version 0.8.x. When using Kafka 0.8.x, use the Kafka Consumer operator, which is compatible with Kafka 0.8.x.

Configuration Parameters

Parameter

Type

Description

maxWaitTime

int

The maximum amount of time in milliseconds that the broker waits for the consumer.

Default: 500

kafkaVersion

string

Mandatory. The version of Kafka installed.

Default: "0.9.0"

brokers

string

Mandatory. Each node in the cluster.

Default: "localhost:9092"

topics

string

Mandatory. The category name to which records are published.

Default: "test_topic"

groupId

string

Mandatory. Holds all information for a consumer that is part of the group.

Default: "test_group"

offset

string

Identifies each record on partition.

Default: "newest"

autoCommit

bool

Specifies whether to automatically commit the message.

Default: true

maxMessageBytes

int

The maximum permitted size of a message.

Default: 1000000

useTLS

bool

Flag that enables using TLS/SSL authentication.

Default: false

insecureSkipVerify

bool

Controls whether a client verifies the server's certificate chain and host name.

Default: false

tlsCAFile

string

Certificate authority file path.

Default: ""

authenticate

bool

Flag that enables SASL authentication.

Default: false

username

string

Username if using kafka's SASL/PLAIN authentication.

Default: ""

password

string

Password if using kafka's SASL/PLAIN authentication.

Default: ""

Input

Input

Type

Description

markOffset

message

Marks the provided message as processed.

Output

Output

Type

Description

message

message

The stream of records produced.