Publishing a Task to OCI Data Flow

Before you publish a task to an application in OCI Data Flow, ensure that you have an Object Storage bucket and the relevant permissions and IAM policies.

See Required Setup and Policies.

1. On the workspace home page, click Projects.
2. Open the project or folder in which the task is saved.
3. On the project or folder details page, click Tasks.
4. In the tasks list, select Publish to OCI Data Flow from the Actions menu of the task that you want to publish.
  
  Only integration and data loader tasks can be published to OCI Data Flow.
5. On the Publish to OCI Data Flow page, complete the Application information section as follows:
  
  Select the compartment in which to create the Data Flow application.
  
  Enter a name and description (optional) for the Data Flow application.
6. Complete the Resource configuration section as follows:
  
  Select a shape for the Spark driver host.
  
  Select a shape for the Spark executor host.
  
  Enter the number of executors to launch when the Data Flow application is run.
7. Complete the Object Storage file configuration section as follows:
  
  Select the Object Storage data asset to use.
  
  Select a connection to the data asset that you have selected.
  
  Select the compartment that has the Object Storage bucket you want to use.
  
  Select the bucket to upload the JAR to.
8. (Optional) Click Show advanced options and select Secure access to private subnet. Then select a private endpoint in OCI Data Flow to use for this application.
9. Click Validate task to check the configuration for any errors that might cause the publish to fail.
10. If validation is successful, click Publish.
  A notification message appears with a View publish status link. Click the link to monitor the publish status in the Oracle Cloud Infrastructure Data Flow publish history panel. See Viewing the OCI Data Flow Publish History for a Task.
Use the oci data-integration external-application create command and required parameters to publish a task to an application in OCI Data Flow:
oci data-integration external-application create [OPTIONS]
For a complete list of flags and variable options for CLI commands, see the Command Line Reference.
Run the CreateExternalPublication operation to publish a task to an application in OCI Data Flow.

Oracle Cloud Infrastructure Documentation

Publishing a Task to OCI Data Flow