Creating a Pipeline Task

• Console
• CLI
• API

• 1. Open the project or folder in which you want to create the task.

     For the steps to open the details page of a project or folder, see Viewing the Details of a Project or Viewing the Details of a Folder.

  2. On the project or folder details page, click Tasks.
  3. In the Tasks section, click Create task and select Pipeline.
  4. On the Create pipeline task page, enter the following basic information:
     1. In the Name and Identifier fields, enter the values that you want or let Data Integration fill in the values automatically, based on the name of the pipeline that you select for this task (in the next step).

        If you want Data Integration to fill in the fields automatically, don't change or enter values before you select a pipeline. You can change the values after the fields are populated with values that are based on the selected pipeline's name.

        In either case, the identifier is a system-generated value based on the name. You can change the value, but after you create and save the task, you can't update the identifier.

     2. (Optional) Enter a description for the task.
     3. Select the Disable simultaneous execution of the task checkbox if you want to disallow concurrent runs of this task.
     4. (Optional) For Project or folder, click Select and select a different project or folder to save the task in.
  5. In the Pipeline section, click Select and select the pipeline that this task runs by following these steps:
     1. In the Select a pipeline panel, perform one of the following actions:
        • Select a pipeline that's saved in the project or folder that you're working in currently.
        • To select a pipeline that's saved in a different project or folder, click Select next to the current project or folder name. In the Select project or folder panel that appears, select the project or folder, and then select the pipeline from the new list of available pipelines.
     2. Click Select.

        Data Integration starts validating the selected pipeline, and you're returned to the Create pipeline task page.

        If there are any errors or warnings in the pipeline, click the pipeline's name to open it in a new tab. Resolve the errors or warnings and save the pipeline. When you navigate back to the Create pipeline task page, Data Integration automatically validates the pipeline again.

  6. To save the task for the first time, click one of the following buttons:

     • Create: Creates and saves the task. You can continue to create and edit the task.

     • Create and close: Creates and saves the task, closes the page, and returns you to the tasks list on the project or folder details page.

  7. Save periodically while you work by clicking one of the following buttons:

     • Save: Commits changes since the last save. You can continue editing after saving.

     • Save and close: Commits changes, closes the page, and returns you to the tasks list on the project or folder details page.

     • Save as: Commits changes (since the last save) and saves to a copy instead of overwriting the current task. You can provide a name for the copy and select a different project or folder for the copy, or save the copy in the same project or folder as the existing task.

  8. (Optional) If parameters are assigned in the selected pipeline, view and change the default parameter values by following these steps:
     1. In the Configure parameters section, click Configure.
     2. On the Configure parameters page, change the default values as needed.

        Consider the following restrictions when editing the default values:

        • For parameterized run properties in tasks that are configured to run in the OCI Data Flow service, use the following format if editing the default parameter values for OCI_DF_LOG_BUCKET and OCI_DF_ARTIFACT_BUCKET: oci://<bucket-name>@<tenancy-name>

        • If the incremental extract strategy for a BICC source is configured to use a date that's managed by the Data Integration system parameter SYS.LAST_LOAD_DATE, you're not allowed to change the date value during design time.

        • If the incremental extract strategy for a BICC source is configured to use a parameterized last extract date that you have added, you can change the date value during design time and runtime.

        • If a parameterized target data entity is configured to use the Merge strategy, you can change the Merge key selection.

        • For a parameterized data asset that requires a staging location: If you select a data asset that doesn't have a default staging location specified in that data asset, Data Integration displays a warning notification. When you see the notification, choose one of the following options:
          • Update that data asset by adding a default staging location.
          • Choose a different data asset that has a default staging location.

        • For a parameterized OCI Data Flow task:

          • With a parameterized driver or executor shape, the OCPUs and memory values are ignored if a non-flexible shape is configured in the parameter. The OCPUs and memory values are used only when you configure a flexible shape as the parameter value.

          • With parameterized OCPUs and memory values for a flexible shape, ensure that you specify a value that's in the acceptable range of values for that property. The acceptable values for OCPUs depend on the flexible shape. The acceptable values for memory depend on the OCPUs value.

     3. Click Configure or Cancel.
        You're returned to the Create pipeline task page.
  9. (Optional) In the Validate task section, click Validate to check the parameter configurations.

     If there are errors or warnings, click View messages. Resolve any errors before you publish the task.

  10. When you finish configuring the task, click Create and close or Save and close.

  Publish the task to an application in Data Integration before you run the task or schedule the task for running. For publishing information, see Publishing to a Data Integration Application.

• Use the oci data-integration task create-pipeline-task command and required parameters to create a pipeline task:

  oci data-integration task create-pipeline-task [OPTIONS]

  For a complete list of flags and variable options for CLI commands, see the Command Line Reference.

• Run the CreateTask operation with the appropriate resource subtype to create a pipeline task.