Support Information and Considerations When Using Pipelines

In Data Integration, a pipeline is a design-time resource for creating a set of tasks connected in a sequence or in parallel to facilitate data processing. It lets you manage and orchestrate the execution of a set of related tasks and activities for a workload instead of running tasks individually and handling run outcomes separately. These pipelines are not designed for low-latency tasks. Occasionally each step might experience several minutes of delay because of network or cloud issues. In between those steps are reconciliation processes that might take a minute or at times even longer.

Ensure that you understand what is supported and any current limitations before you create pipelines in Data Integration.

• By default, a workspace has a limit of four concurrent task runs. When you have more than four tasks in a pipeline, integration tasks and data loader tasks are queued based on the limit of four concurrent runs per default workspace. SQL, OCI Data Flow, and REST tasks are not queued.
• The limit of four concurrent runs is across pipelines in a workspace; the limit is not just within a pipeline.
• When designing a pipeline, remember that while the maximum supported concurrency is 16, only four concurrent task runs can take place at a time during execution.
• Nesting of pipelines is supported, up to a maximum of three.
• We don't support running more than 100 tasks in a single pipeline. This limit includes the tasks in nested pipelines.
• In a pipeline, when an operator's output parameters are used in the next operator, refrain from using the same output parameter multiple times as doing so might result in indeterminate result.