Oracle Cloud Infrastructure Documentation
Try Free Tier

Harvest from Autonomous Databases with Private Access

Harvesting is a process that extracts technical metadata from your data source into your data catalog. This tutorial provides the steps to harvest from a data source that is only accessible privately.

In this tutorial, you:

  1. Create the policies needed to harvest an autonomous database with private URL.
  2. Obtain the autonomous database access details.
  3. Create a private endpoint in Data Catalog.
  4. Attach the private endpoint to your data catalog.
  5. Create a data asset.
  6. Add a connection for the data asset.
  7. Harvest the data asset.

For more information, see configuring a private network.

Before You Begin

To successfully perform this tutorial, you must have the following:

If you already have the autonomous database you want to harvest from, you can use the details for that database to complete this tutorial. If you don't have an existing autonomous database with private access and want to try this tutorial, you can follow the following instructions to set up the resources needed to perform this tutorial.

Setting up the Resources Needed for this Tutorial

1. Create Access Policies

To configure Data Catalog to access the private network of a data source, you need access to networking and data catalog resources.

If you already have access to perform all Data Catalog and Networking operations in your required compartments, you may skip this step.

To create the policy needed to configure a private network in data catalog, perform the following steps:

  1. Open the navigation menu  and select Identity & Security. Under Identity, select Policies.
  2. In the Policies page, click Create Policy.
  3. In the Create Policy panel, enter the following details:
    • Name: Enter a unique name for the policy. The name must be unique across all policies in your tenancy. You can't change the name later. For example, data-catalog-private-endpoint-policy.
    • Description: Enter a description, such as Grant permissions to create private endpoint.
    • Compartment: Select a compartment in which you want to create the policy.
    • Policy Builder: In this section, move the slider to Show manual editor, and enter the policy rule. For example, for the data-catalog-users group, enter the following policy rule:
      allow group data-catalog-users to manage data-catalog-private-endpoints in tenancy
      Note

      This policy allows users in the data-catalog-users group to perform all data catalog private endpoint operations in any compartment in the tenancy.
    • Select Create Another Policy and enter the following policy rule: 
      allow group data-catalog-users to manage virtual-network-family in tenancy
      Note

      This policy allows users in the data-catalog-users group to perform all network-related operations in any compartment in the tenancy.
  4. Click Create.
You have successfully created the policies to access the required resources for configuring a private network in Data Catalog.

2. Obtain Data Source Details

You need the private network and database connection information for the autonomous database you want to harvest.

Obtain the following details for the autonomous database:

Information Needed Instructions to Obtain Information
For configuring the private network, you need the VCN and subnet name and the private URL of the database.
  1. From the navigation menu in the Console, click Autonomous Data warehouse.
  2. View the details for the database you want to harvest.
  3. From the Network section, note the VCN, Subnet, and Private Endpoint URL of the database.

Note: If you have more database in this network (same VCN and subnet) that you want to harvest, make note of the Private URL of those databases too.
For creating the data asset, you need the database name. From the autonomous database details page, note the database name from the General Information section.
For adding a connection, you need the database wallet and sign in credentials.
  1. From the autonomous database details page, click DB Connection.
  2. Click Download Wallet.
  3. Enter a password for this wallet. You don't use this password in this tutorial.
  4. Click Download.
  5. Save the wallet file in your local machine.

You also need the credentials (username and password) for the database that you specified when you created the autonomous database. If you didn't create the autonomous database, obtain the credentials from your admin. While harvesting, you can view only the database entities you have access to.

3. Create a Private Endpoint

You create a Data Catalog private endpoint to configure the network access details for the autonomous database data sources you want to harvest.

To create a private endpoint in Data Catalog, perform the following steps:

  1. Open the navigation menu  and select Analytics & AI. Under Data Lake, select Data Catalog.
  2. Click Private Endpoints. The Private Endpoints page displays.
  3. Click Create Private Endpoint. A Create Private Endpoint panel displays.
  4. Ensure you have permission to work in the selected compartment, and enter a name for the private endpoint. For example, XYZ Private Endpoint.
  5. Select the VCN and subnet where the autonomous database is hosted.
  6. Enter the DNS zone (Private Endpoint URL) for the autonomous database. Use a comma to enter more than one data source private URL.
    Note

    To view autonomous database Private Endpoint URL in the Console, from the navigation menu click Autonomous Database and view the details for your autonomous database. The Private URL is displayed under Network.
  7. Click Create.
The private endpoint is being created. The create process can take a couple of minutes. When the private endpoint is created successfully, the private endpoint is in ACTIVE status.

If the private endpoint status changes to FAILED, ensure you have the created the access policies and set up your private network correctly.

4. Attach a Private Endpoint

You attach a private endpoint to a data catalog to allow data assets to be created for data sources available in the private network.

To attach a private endpoint to a data catalog, perform the following steps:

  1. Click Data Catalogs.
  2. Click the Actions menu for the data catalog where you want to attach the private endpoint and select Attach Private Endpoint.
  3. Select the private endpoint you created in the previous step and click Attach.
The data catalog status changes to Updating, and the private endpoint is being attached. After the private endpoint is attached successfully, the status of the data catalog changes to Active.

5. Create an Autonomous Database Data Asset

You are now ready to register your private IP autonomous database with Data Catalog as a data asset . In this tutorial, you create an Autonomous Data Warehouse data asset.

To create an autonomous database data asset, perform the following steps:

  1. Click the data catalog instance where you attached the private endpoint in the previous step.
  2. From your data catalog Home tab, click Create Data Asset from the Quick Actions tile.
  3. In the Create Data Asset panel, enter a name to uniquely identify your data asset. Optionally, enter a description.
  4. From the Type list, select Autonomous Data Warehouse.
  5. In the Database Name field, enter the database name you specified when you created the autonomous database.
    Note

    To view autonomous database name in the Console, from the navigation menu click Autonomous Database and view the details for your autonomous database. The database name is displayed under General Information.
  6. Select the Use private endpoint check box.
  7. Click Create.
You have successfully created your autonomous database data asset.

6. Add a Connection

After you register a data source as a data asset in your data catalog, you create a connection to your data asset to harvest it. You can create multiple connections to your data source. At least one connection is needed to harvest a data asset.

For autonomous database data source types, you can use secrets in Oracle Cloud Infrastructure Vault to store the password that you need to connect to the source using a connection. By using OCI Vault, you provide the OCID of the secret when specifying the connection details, so you don't have to enter the actual password when you create the data asset. You can also use secrets for the Oracle wallet and passwords instead of uploading the wallet when you create your data asset.

A vault is a container for keys and secrets. Secrets store credentials such as required passwords for connecting to data sources. You use an encryption key in a vault to encrypt and import secret contents to the vault. Secret contents are based64-encoded. Data Catalog uses the same key to retrieve and decrypt secrets while connecting a data asset to the data source. For more information about vault, key, and secret, see Overview of Vault. For information about copying the secret OCID, see View Secret Details.

To use an Oracle wallet with secrets in OCI Vault, you must:
  • Provide a wallet password when you download the wallet.
  • Remove the .p12 file from the downloaded wallet zip.
  • Use any base64 encoder to encode the modified wallet zip to base64.
  • Copy the base64-encoded data to a secret in a vault.
  • Create a secret for the database password.

To add a connection for the autonomous database data asset, follow these steps:

  1. On the Home tab, click Data Assets.
  2. In the Data Assets list, select the autonomous database data asset that you created.
  3. On the data asset details page, under Summary, in the Connections section, click Add Connection.
  4. In the Add Connection panel, enter the details as described in the following table:
    Field Description
    Name Enter a unique name for your connection.
    Description Enter a short description for your connection.
    Type Select Generic.
    Use Wallet Select this option to upload the file with the client credentials that's downloaded from the autonomous database. When you select this option, the following fields appear:
    • Wallet - Upload the file with the client credentials that is downloaded from your autonomous database. For more information, see downloading client credentials.
    • TNS Alias - From the TNS Alias drop-down list, select a TNS Alias. For better performance, select the <name>_low option.
    • User Name - Enter the admin user name that is set when the autonomous database is created.
    • Password - Enter the password for the autonomous database admin user name.
    Use Vault Secret OCID Select this option to enter the OCID of the secret that is created in OCI Vault for the client credentials file and file password. When you select this option, the following fields appear:
    • Vault Secret OCID for Wallet - Enter the OCID of the secret that is created in OCI Vault for the wallet. For information about copying the secret OCID, see View Secret Details.
    • User Name - Enter the admin user name that is set when the autonomous database is created.
    • Vault Secret OCID for Password - Enter the OCID of the secret that is created in OCI Vault for password for the autonomous database admin user name.
    Make this the default connection for the data asset. Select this check box to make this connection the default connection for the data asset.
    Test Connection Click the button to test your connection.
  5. Click Add.

7. Harvest the Data Asset

You're now ready to harvest your autonomous database data asset. Your autonomous database must have the data from which you want to harvest the technical metadata. If you used the setup instructions in this tutorial, you can harvest metadata from the default data that's available in your autonomous database.

To harvest your autonomous database data asset, perform the following steps:

  1. Click Harvest on the data asset details page of a data asset.
  2. The Select Connection page displays and the default connection is selected. Click Next.
  3. The Select Data Entities page displays. View and add all the data entities you want to harvest from the Available ADW Schema section.
    1. Click the add icon for each data entity you want to include in the harvest job.
    2. Click Add All to select all the entities for harvesting.
    3. Use the Filter ADW Schema box to find a data entity from the available data entities.
    4. Use the page navigation icons to browse all the data entities.
    5. Click the remove icon for any selected data entity that you want to remove from the harvest job.
    6. If you need to start over, click Remove All and then start over.
    After you have reviewed the data entities you want to harvest from the Selected ADW Schema / Data Entities section, click Next.
  4. The Create Job page displays. In the Job Name field, enter a unique name to identify the harvest job.
  5. Optionally, enter a Description.
  6. Select Run job now and then click Create Job.
  7. The job to harvest your autonomous database data asset is created successfully and the Jobs tab displays. To view job details, click the job name.
Your data asset is harvested successfully and you can review the harvest job details.

Was this article helpful?