Oracle Cloud Infrastructure Documentation
Try Free Tier

Setting Up Data Sources

Register the data sources that you want to harvest, in Data Catalog. You register the data sources as data assets.

You create a data asset from a wide range of supported data sources. To set up and manage your data assets, see:

Required IAM Policies for Data Integration Data Asset

Before creating an OCI Data Integration data asset, set the following policies to create, harvest Data Integration data asset and view the lineage for it.

As a prerequisite, create a dynamic group that includes the specific Data Catalog OCID as a resource in the group.

Any {resource.id = '<catalog_ocid>'}

  • Set the following policies to allow Data Catalog instance to read the Data Integration workspaces and harvest lineage metadata from them:

    Allow dynamic-group <dynamic-group-name> to read dis-workspaces in compartment <DIS Workspace Compartment>
    Allow dynamic-group <dynamic-group-name> to read dis-workspaces-lineage in compartment <DIS Workspace Compartment>
    Note

    If you are using the following Service Principal-based policies, you must modify them to the policies mentioned in the previous paragraph by February 28, 2024:
    Allow dynamic-group <dynamic-group-name> to use dis-workspaces in compartment <DIS Workspace Compartment>
    Allow service datacatalog to {DIS_WORKSPACE_LINEAGE_INSPECT, DIS_WORKSPACE_OBJECT_INSPECT} in compartment <DIS Workspace Compartment Name>

  • Viewing lineage requires CATALOG_DATA_ASSET_READ permission.

    Allow <any-user> to read data-catalog-data-assets in compartment <compartment name> where target.catalog.id='<Data Catalog OCID>'

Required IAM Policies for Data Flow Data Asset

Before creating a data asset in the catalog for OCI Data Flow in a different tenancy, set the following policies.

In the Data Catalog tenancy, set:

admit any-user of tenancy <Data Flow Tenancy> to manage data-catalog-data-assets in tenancy where all {request.principal.type = 'dataflowrun'}

In the Data Flow tenancy, set:

endorse any-user to manage data-catalog-data-assets in tenancy <Data Catalog Tenancy> where all {request.principal.type = 'dataflowrun'}

Set the following policy when the Data Flow application and the Data Catalog are in the same tenancy: 

allow any-user to manage data-catalog-data-assets in tenancy where all {request.principal.type = 'dataflowrun'}

Required IAM Policies for Metastore

Before you create Metastore data assets, you create policies to enable access to the required data objects.

As a prerequisite, create a dynamic group that includes the specific data catalog OCID as a resource in the group. 
Any {resource.id = ‘<catalog_ocid>’ }

Required IAM Policies for MySQL Data Asset

You can use the OCI MySQL Database service as a data source to create a MySQL data asset. In this section, learn about the policies that you need to for a group named Administrators, to work with the MySQL Database service.

Create this policy to allow members of the group Administrators to list and read the contents of all compartments in the tenancy:
Allow group Administrators to {COMPARTMENT_INSPECT} in tenancy
Create this policy to allow members of the Administrators group to read, attach, and detach subnets and read Virtual Cloud Networks (VCNs) in the tenancy:
Allow group Administrators to {VCN_READ, SUBNET_READ, SUBNET_ATTACH, SUBNET_DETACH} in tenancy
Note

You need access to these resource types to attach a DB System to a VCN.
Create this policy to allow members of the Administrators group access to all aspects of MySQL Database Service in the tenancy:
Allow group Administrators to manage mysql-family in tenancy
Create this policy to harvest data from MySQL database:
allow group <your-group-name> to use mysql-instances in tenancy

Required IAM Policies for Oracle Object Storage Data Asset

Before you create Oracle Object Storage data assets, you create policies to enable access to the required data objects. After creating these policies, when you harvest the Object Storage data asset, only those data entities that your data catalog instance has access to are listed. You can select the data objects you want to harvest from the displayed list.

At the least, you must have READ permission for all the individual resource types objectstorage-namespaces, buckets, and objects, or for the Object Storage aggregate resource type object-family. For step-by-step instructions, see tutorial Harvesting from Oracle Object Storage.

Create Resource Principal policies to allow Data Catalog access objects in Object Storage. As a prerequisite, create a dynamic group that includes the specific data catalog OCID as a resource in the group.

Example:

Any {resource.id = 'ocid.datacatalog.oc1..<unique_ID>'}

Required IAM Policies for using OCI Vault

To use Oracle Cloud Infrastructure Vault, you create a dynamic group that includes the data catalog service as a resource in the group. 

Any {resource.type='datacatalog' }
Create the following policy to allow the dynamic group to read the secret:
Allow dynamic-group <dynamic group name> to read secret-family in tenancy
To use an Oracle wallet with secrets in OCI Vault, you must:
  • Provide a wallet password when you download the wallet.
  • Remove the .p12 file from the downloaded wallet zip.
  • Use any base64 encoder to encode the modified wallet zip to base64.
  • Copy the base64-encoded data to a secret in a vault.
  • Create a secret for the database password.

Was this article helpful?