Sizing for Data Pipelines

Sizing of Data Pipelines support a flow-based programming paradigm for data ingestion and transformation.

The cloud resource requirements for data pipelines are determined as the sum of the individual requirements for each of the concurrent pipelines at peak times. The number of concurrent data pipelines influences resource requirements because each pipeline consumes resources. Simply spoken, the more pipelines are running at a given point in time, the more memory and CPUs are required. If pipelines are executed sequentially, the requirements per pipeline do not add up, but the system needs to have enough memory and CPUs for the most "demanding" one.

The range of possibilities for data pipelines is very large, and resource requirements may differ a lot. It is strongly advisable to measure the memory and CPUs required for your pipelines early and incorporate the learnings into sizing of your (production) environment.

Therefore, this section is structured as follows:

1. Discuss the characteristics of data pipelines that influence the sizing.

2. Define conditions under which a simplified sizing calculation is sufficient, in terms of a rough estimate, and provide the formula of this estimate.

3. For the more complex cases, you can learn the procedure for how to estimate the resource requirements of a more demanding pipeline. This procedure requires that a test environment for SAP Data Intelligence Cloud is already available.