Ingest Google BigQuery SQL
Google BigQuery SQL Consumer is based on datahub-flowagent and uses Data Services as execution engine.
SQL Consumer takes in native SQL statement that you would run on Google BigQuery as input, and it will read the response result set from the source piece-wise and load to the next operator in line.
Components
- Triggering the graph:
constantgenerator1: Provides constant input of SQL text to Consumer which also triggers the ingestion.
- Defining connection:
googlebigquerysqlconsumer1: Provides consumer connection information.
- Loader definition:
flowagentfileproducer1: Data Services requires a loader, such as CSV Loader, to load to Vflow operators.