Modeling Guide

Google Pub/Sub Consumer

This operator consumes publications from a Google Pub/Sub topic and outputs them as a stream of messages.

  • If Subscription name refers to an existing subscription name on the Google Pub/Sub service, Topic can be empty, as an existing subscription is already attached to a topic. If subscription is to be created, Topic must be provided and Topic must exist on the pub/sub service. Otherwise, the graph fails regardless of the value of Fail on error.

  • If Fail on error is false, upon error, the consumer only outputs the error message on Error output port and does not fail the graph.

  • The messageToAck input port is optional. If connected, no automatic acknowledgment is sent and the user has to explicitly acknowledge received publications by submitting their message ID to this port. This allows using the Google Pub/Sub service as a message queue and ensures that publications that are not consumed successfully get delivered again to the same consumer or other consumers using the same Subscription name.

  • When using explicit acknowledgment, receiving more than one ack for the same publication or requesting acknowledgment for a publication that is not received by the consumer, results in an error.

Configuration Parameters

Parameter

Type

Description

Connection

string

Mandatory: A Google Pub/Sub connection consisting of a GCP project ID and a JSON key file to access the Pub/Sub service.

Subscription name

string

Mandatory: The name of the subscription on the Google Pub/Sub service.
Topic

string

Google Pub/Sub topic name.
Fail on error

boolean

Whether to fail the whole graph if the consumer encounters an error at runtime.

Default: false

Maximum acknowledgment time in seconds integer Maximum time period between when the consumer receives the message and it sends the acknowledgment to the Google Pub/Sub service. If not acknowledged within this period, the Pub/Sub service delivers the publication again to the same consumer or any other consumer which is configure to use the same Subscription name. If the messageToAck input port is not used, the consumer automatically sends the acknowledgment for a message upon receiving it from the pub/sub service.

Default: 600

Maximum outstanding publications integer Maximum number of received but unacknowledged messages on the consumer. Upon reaching this maximum, the consumer stops pulling publications form the pub/sub service until one or more of the existing messages expire (ack deadline is passed) or acknowledged. A negative value indicates no limit for the number of unprocessed messages.

Default: 1000

Input

Input

Type

Description

messageToAck

message

The stream of Message IDs. Each publication received by the consumer has a unique ID assigned by the Google Pub/Sub service. Using this ID, the publication can be explicitly acknowledged. The operator reads the message ID from the attribute named msg.MessageID in the received message.

Output

Output

Type

Description

receivedMessage

message

The stream of publications received by the consumer from the Topic on Google Pub/Sub service. Each output message contains the following extra two attributes:
  • gcp.pubsub.message.ID: a unique ID assigned to each message published to Google Pub/Sub. This ID can be used to explicitly acknowledge a received publication.

  • gcp.pubsub.message.publishTime: The timestamp assigned to the message by the Google Pub/Sub service upon its publication.

error

message

Error messages occurring during runtime are output on this port as a stream of messages (one for each error). The error message has the attribute msg.KeyError with the value true and the body of the message is the error message.