Overview of Building Pipelines
Use the Data Integration pipeline designer to build a pipeline.
The intuitive UI pipeline designer is similar to the designer that you use to create a data flow. To design a pipeline, you use components called operators that represent published tasks and design-time tasks, and other processing activities. You can use Data Integration built-in functions with the expression and decision operators in a pipeline to build expressions that process data.
If parameters are assigned in the tasks or in the associated data flows wrapped by integration tasks, you can override the default values by configuring incoming parameter values directly during design time, and also when you run the pipeline by using user-defined and system parameters.
By using a pipeline, you can build a complex data graph and automate an entire workload of interdependent processes and tasks. For example, you can leverage a pipeline to load data from multiple source formats in parallel, apply data transformations, and merge the parallel tasks so that downstream tasks can further process the data based on the outcomes of the upstream operations.
Create pipelines from a project or folder details page. The Pipelines section of a project or folder details page shows the list of pipelines created in that project or folder. To quickly find a pipeline, enter the complete name of a pipeline in the Filter by name field, and press Enter. You can also use the Favorite filter menu to display only those pipelines that are marked as favorite. Clicking Clear removes the filter.
From the list of pipelines, you can perform the following actions on a pipeline by using the Actions menu (
):
- View details: Displays the pipeline for editing.
- Copy pipeline key: Copies to the clipboard the key of the pipeline, which you can use with Data Integration API.
- Duplicate: Lets you make a copy of the pipeline.
- Move: Lets you move the pipeline from its current location to another project or folder.
- Delete: Lets you delete the pipeline. After a pipeline is deleted, it cannot be restored.
After building a pipeline, create a pipeline task that's used to run the orchestration.