Oracle Cloud Infrastructure Documentation

Data Asset Properties for Data Sources

To create a data asset, enter the required details depending on the type of data source.

Apache Hive Data Asset

Create an Apache Hive data asset by entering the following details in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select Apache Hive.
  • Host: Enter the DNS name or the public or private IP address of your database. For example, mydatabase.com or 192.0.2.1.

    If your Apache Hive is configured in a private network, then you can specify the Fully Qualified Domain Name (FQDN) or the private IP for the database.

    FQDN example in Oracle Cloud Infrastructure: 
    <hostname>.<subnet DNS label>.<VCN DNS label>.oraclevcn.com
  • Port: Enter the port number opened for accessing your database on the host specified. Ensure the port that you specify has the security rule already set up. See about security lists and how to create a security list.
  • Database: Enter the database name for your Apache Hive.
  • Transport Mode: Select one of the following options:
    • HTTP - If you select this option, the HTTP Path field appears. Enter the path value.
    • Binary
  • Enable SSL: Select this check box to enable SSL. If you select this check box, then in the Host field, you must enter the Fully Qualified Domain Name (FQDN) or the private IP for the data source.
  • Enable kerberos: Select this check box to enable Kerberos. When you select this check box, the KDC field appears. Enter the key distribution center details in the KDC field.
  • Use private endpoint: Select this check box, if your data asset is hosted in a private network. You must have already created and attached a private endpoint to your data catalog. If no private endpoint is attached to the data catalog, you receive an error while creating the data asset.

Apache Kafka

Create an Apache Kafka data asset by entering the following details in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select Kafka.
  • Bootstrap servers: Enter the host and port pair details for the Kafka cluster. For example, <hostname>:<portnumber>.

    You can enter more than one pair by using a comma separator. For example, <hostname>:<portnumber>, <anotherhostname>:<portnumber> .

    In the Kafka server's server.properties file (usually available in the config folder of the Kafka server installation directory), the following properties are commented out by default:
    • #listeners = plaintext://your.host.name:9092]
    • #advertised.listeners = plaintext://your.host.name:9092]

    To harvest a Kafka data asset, you must ensure that the advertised.listeners property is uncommented and points to a public hostname or IP address (not a private IP address).

  • Use private endpoint: Select this check box, if your data asset is hosted in a private network. You must have already created and attached a private endpoint to your data catalog. If no private endpoint is attached to the data catalog, you receive an error while creating the data asset.

Autonomous Database

Data Catalog supports metadata harvest from Autonomous Database (ADB), which can be configured using one of the following ways:
  • Access Control List (ACL)
  • Private endpoint
For more information about the different configurations of ADB, see Configure Network Access with Access Control Rules (ACLs) and Private Endpoints.
For an ADB configured with ACL, access for Data Catalog within a region requires the following details:
  • Oracle services must communicate with each other through service gateway privately. For more information, see Access to Oracle Services: Service Gateway.
  • Service gateway must be configured in VCN where the ADB instance is created.
  • Configure ACL with CIDR 240.0.0.0/4 in ADB

For an ADB configuration with ACL, Data Catalog access across regions isn't possible as the IP/CIDR provided in ACL must be public; the Data Catalog CIDR isn't public.

For an ADB configured with private endpoint, access for Data Catalog within a region requires the following details:
  • Create a private endpoint with VCN and subnet of Autonomous Data Warehouse (ADW).
  • Attach the private endpoint to the Data Catalog instance so that the catalog can access ADW.
Create an Autonomous Database data asset by entering the following details in the Create Data Asset panel.
  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select Autonomous Data Warehouse or Autonomous Transaction Processing.
  • Database Name: Enter the name of your autonomous database.
  • Use private endpoint: Select this check box, if your data asset is hosted in a private network. You must have already created and attached a private endpoint to your data catalog. If no private endpoint is attached to the data catalog, you receive an error while creating the data asset.

Data Flow

Create an OCI Data Flow data asset only for the Data Flow service in a different tenancy to harvest and view the lineage of the data processed in it.

Note

A data asset is automatically created for Data Flow in the same tenancy as the catalog instance. It's created the first time lineage is pushed from a Data Flow application to the catalog.

Before you create a Data Flow data asset, see Required IAM Policies for Data Flow Data Asset. Provide the following information in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select OCI Data Flow.
  • Tenancy OCID: Enter the tenancy OCID where the data asset will be located.

After the data asset for the remote Data Flow service is created, any lineage metadata generated by applications running in that service is pushed to this catalog instance. You must ensure that the Data Flow applications in the remote tenancy are configured to generate lineage metadata and the required IAM policies are set up in both tenancies. You can view this newly created data asset in the data assets tab. See Data Lineage Overview.

Data Integration

You can create an OCI Data Integration data asset for Data Integration workspace to harvest and view the lineage of the data processed in it.

Before you create Data Integration data asset, see Required IAM Policies for Data Integration Data Asset. Provide the following information in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select OCI Data Integration.
  • Compartment: Select the compartment.
  • Workspace: Select the workspace.
  • Required policies for this data asset: This information box displays the required policies for the Data Integration data asset. To copy the policies, click Copy. To add the policies, contact your administrator.

After the data asset for the Data Integration workspace is created, configure a sync job that runs regularly to pull the lineage information from the workspace. The default frequency of the sync is one hour. You can disable the sync by clicking the Disable sync button that appears in the details page of the data asset. If you disable sync, Data Catalog does not fetch lineage information from that Data Integration workspace until it is enabled again. To modify the default hourly sync frequency, disable and re-enable sync to specify the new frequency. You can view this newly created data asset in the data assets tab. See Data Lineage Overview.

IBM DB2

Create an IBM DB2 data asset by entering the following details in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select IBM DB2.
  • Host: Enter the DNS name or the public or private IP address of your database. For example, mydatabase.com or 192.0.2.1.

    If your IBM DB2 is configured in a private network, then you can specify the Fully Qualified Domain Name (FQDN) or the private IP for the database.

    FQDN example in Oracle Cloud Infrastructure: 
    <hostname>.<subnet DNS label>.<VCN DNS label>.oraclevcn.com
  • Port: Enter the port number opened for accessing your database on the host specified. Ensure the port that you specify has the security rule already set up. See about security lists and how to create a security list.
  • Database: Enter the database name for your IBM DB2.
  • Use private endpoint: Select this check box, if your data asset is hosted in a private network. You must have already created and attached a private endpoint to your data catalog. If no private endpoint is attached to the data catalog, you receive an error while creating the data asset.

Metastore

Create a Metastore data asset by entering details in the the required details.

Before creating Metastore data asset, see Required IAM Policies for Metastore. Provide the following information in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select Metastore.
  • Compartment: Select the compartment that contains the metastore from which the data asset is created.
  • Metastore: Select the metastore from which the data asset is created.

Microsoft Azure SQL Database

Create a Microsoft Azure SQL data asset by entering the following details in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select Microsoft Azure SQL Database.
  • Host: Enter the DNS name or the public or private IP address of your database. For example, mydatabase.com or 192.0.2.1.

    If your Microsoft Azure SQL Database is configured in a private network, then you can specify the Fully Qualified Domain Name (FQDN) or the private IP for the database.

    FQDN example in Oracle Cloud Infrastructure: 
    <hostname>.<subnet DNS label>.<VCN DNS label>.oraclevcn.com
  • Port: Enter the port number opened for accessing your database on the host specified. Ensure the port that you specify has the security rule already set up. See about security lists and how to create a security list.
  • Database: Enter the database name for your Microsoft Azure SQL database.
  • Use private endpoint: Select this check box, if your data asset is hosted in a private network. You must have already created and attached a private endpoint to your data catalog. If no private endpoint is attached to the data catalog, you receive an error while creating the data asset.

Microsoft SQL Server

Create a Microsoft SQL Server data asset by entering the following details in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select Microsoft SQL Server.
  • Host: Enter the DNS name or the public or private IP address of your database. For example, mydatabase.com or 192.0.2.1.

    If your Microsoft SQL Server database is configured in a private network, then you can specify the Fully Qualified Domain Name (FQDN) or the private IP for the database.

    FQDN example in Oracle Cloud Infrastructure: 
    <hostname>.<subnet DNS label>.<VCN DNS label>.oraclevcn.com
  • Port: Enter the port number opened for accessing your database on the host specified. Ensure the port that you specify has the security rule already set up. See about security lists and how to create a security list.
  • Database: Enter the database name for your Microsoft SQL Server database.
  • Use private endpoint: Select this check box, if your data asset is hosted in a private network. You must have already created and attached a private endpoint to your data catalog. If no private endpoint is attached to the data catalog, you receive an error while creating the data asset.

MySQL

Create a MySQL data asset and add connection by entering the following details in Create Data Asset panel.

Before creating MySQL data asset, see Required IAM Policies for MySQL Data Asset.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select MySQL.
  • Host: Enter the DNS name or the public or private IP address of your database. For example, mydatabase.com or 192.0.2.1.

    If your MySQL database is configured in a private network, then you can specify the Fully Qualified Domain Name (FQDN) or the private IP for the database.

    FQDN example in Oracle Cloud Infrastructure: 
    <hostname>.<subnet DNS label>.<VCN DNS label>.oraclevcn.com
  • Port: Enter the port number opened for accessing your database on the host specified. Ensure the port that you specify has the security rule already set up. See about security lists and how to create a security list.
  • Database: Enter the database name for your MySQL database.
  • Use private endpoint: Select this check box, if your data asset is hosted in a private network. You must have already created and attached a private endpoint to your data catalog. If no private endpoint is attached to the data catalog, you receive an error while creating the data asset.

Oracle Database

Create an Oracle Database data asset by entering the following details in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select Oracle Database.
  • Host: Enter the DNS name or the public or private IP address of your database. For example, mydatabase.com or 192.0.2.1.

    If your Oracle Database is configured in a private network, then you can specify the Fully Qualified Domain Name (FQDN) or the private IP for the database.

    During data asset creation, we recommend using Fully Qualified Domain Name (FQDN).

    FQDN example in Oracle Cloud Infrastructure:
    <hostname>.<subnet DNS label>.<VCN DNS label>.oraclevcn.com
  • Port: Enter the port number opened for accessing your database on the host specified. Ensure the port that you specify has the security rule already set up. See about security lists and how to create a security list.
  • Database: Enter the database name or SID for your Oracle Database.
  • Use private endpoint: Select this check box, if your data asset is hosted in a private network. You must have already created and attached a private endpoint to your data catalog. If no private endpoint is attached to the data catalog, you receive an error while creating the data asset.
  • RAC enabled: Select Use private endpoint to view RAC enabled check box. Select this check box if your Oracle Database (including Exadata) is RAC enabled. This creates a scan proxy resource for the first time, when you run Test Connection or create a harvest job for this data asset. An asynchronous job starts running to create the scan proxy resource. You must monitor the job and wait until the job completes before rerunning Test Connection or creating a harvest job. See Viewing Job Details.

Oracle Object Storage

Create an Oracle Object Storage data asset by entering the following details in the Create Data Asset panel.

Before creating Oracle Object Storage data asset, see Required IAM Policies for Oracle Object Storage Data Asset.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select Oracle Object Storage.
  • URL: Enter the swift URI for your Oracle Object Storage resource in the following format: https://swiftobjectstorage.<region-identifier>.<realm_domain>
  • Namespace: Enter the object storage namespace for the specified Oracle Cloud Infrastructure resource. To view your Object Storage namespace string in the Console, from the Profile menu click Tenancy:<your_tenancy_name>. The namespace is listed under Object Storage Settings.
  • Required policies for this data asset: This information box displays the required policy for the Object Storage data asset. To copy the policy, click Copy. To add the policy, contact your administrator.

PostgreSQL

Create a PostgreSQL data asset by entering the following details in the Create Data Asset panel.

  • Name: Enter a name to uniquely identify your data asset. You can edit the name later.

    You can't use the following special characters in the name: & < > " ' / \ = ;

    Name is a searchable field in Data Catalog.

  • Description: Specify the need or purpose for creating this data asset.
  • Type: Select PostgreSQL.
  • Host: Enter the DNS name or the public or private IP address of your database. For example, mydatabase.com or 192.0.2.1.

    If your PostgreSQL database is configured in a private network, then you can specify the Fully Qualified Domain Name (FQDN) or the private IP for the database.

    FQDN example in Oracle Cloud Infrastructure: 
    <hostname>.<subnet DNS label>.<VCN DNS label>.oraclevcn.com
  • Port: Enter the port number opened for accessing your database on the host specified. Ensure the port that you specify has the security rule already set up. See about security lists and how to create a security list.
  • Database: Enter the database name of your PostgreSQL database.
  • Use private endpoint: Select this check box, if your data asset is hosted in a private network. You must have already created and attached a private endpoint to your data catalog. If no private endpoint is attached to the data catalog, you receive an error while creating the data asset.