Creating a Pipeline
Create a Data Science pipeline to run a task.
Ensure that you have created the necessary policies, authentication, and authorization for pipelines.
For proper operation of script steps, ensure that you have added the following rule to a dynamic group policy:
all {resource.type='datasciencepipelinerun', resource.compartment.id='<pipeline-run-compartment-ocid>
'}
Before you begin:
-
Review the use of pipelines environment variables.
-
To store and manage pipeline logs, learn about logging.
-
Decide which conda environment you want to use. If you need a custom conda environment, create and publish one.
You can create pipelines by using the ADS SDK, OCI Console, or the OCI SDK.
Using ADS for creating pipelines can make developing the pipeline, the steps, and the dependencies easier. ADS supports reading and writing the pipeline to and from a YAML file. You can use ADS to view a visual representation of the pipeline. We recommend that you use ADS to create and manage pipeline using code.
- From a Job
The pipeline step uses an existing job. Select one of the jobs in the tenancy.
From a ScriptThe step uses a script to run. You need to upload the artifact containing all the code for the step to run.
From ContainerOptionally, when defining pipeline steps, you can choose to use Bring Your Own Container. These environment variables control the pipeline run.
You can use the OCI CLI to create a pipeline as in this Python example:
The ADS SDK is also a publicly available Python library that you can install with this command:
pip install oracle-ads
You can use the ADS SDK to create and run pipelines.
Custom Networking
Use a custom Network that you've already created in the pipeline to give you extra flexibility on the network.
Creating Pipelines with Custom Networking
You can choose to use custom networking when creating a pipeline.
Switching from custom networking to managed networking isn't supported after the pipeline is created.
If you see the banner:
The specified subnet is not accessible. Select
a different subnet.
create a network access policy as described in the section,
Pipeline Policies.Using the Console
Choose to use custom networking in the Create pipeline panel.
If you choose default networking, the system uses the existing service-managed network. If you select the custom networking option, you're prompted to pick a VCN and a subnet.
Select the VCN and subnet that you want to use for the resource. For egress access to the public internet, use a private subnet with a route to a NAT gateway. If you don't see the VCN or subnet that you want to use, click Change Compartment, and then select the compartment that contains the VCN or subnet.
Using APIs
subnet-id
in the
infrastructure-configuration-details
to use a custom subnet on the
pipeline level. For example:"infrastructure-configuration-details": {
"block-storage-size-in-gbs": 50,
"shape-config-details": {
"memory-in-gbs": 16.0,
"ocpus": 1.0
},
"shape-name": "VM.Standard.E4.Flex",
"subnet-id": "ocid1.subnet.oc1.iad.aaaaaaaa5lzzq3fyypo6x5t5egplbfyxf2are6k6boop3vky5t4h7g35xkoa"
}
"step-infrastructure-configuration-details": {
"block-storage-size-in-gbs": 50,
"shape-config-details": {
"memory-in-gbs": 16.0,
"ocpus": 1.0
},
"shape-name": "VM.Standard.E4.Flex",
"subnet-id": "ocid1.subnet.oc1.iad.aaaaaaaa5lzzq3fyypo6x5t5egplbfyxf2are6k6boop3vky5t4h7g35xkoa"
},