Harvest Oracle Database Hosted in OCI Public Subnet Using Private Endpoint
Oracle Database systems are protected with network security rules that restrict the network traffic to only authorized subnets and IPs. Therefore, you must create and configure a private endpoint so thatData Catalog can connect to the database system.
In this tutorial, you do the following:
- Create the policies needed to harvest from Oracle database systems using private endpoint.
- Obtain the Oracle database system access details.
- Create a private endpoint in Data Catalog.
- Attach the private endpoint to your data catalog.
- Create a data asset.
- Harvest the data asset.
For more information, see configuring a private network.
Before you Begin
- An Oracle Cloud Infrastructure account. See signing up for Oracle Cloud Infrastructure.
- Access to use the Data Catalog resources. See getting started and policy examples.
- A created data catalog instance. See creating a data catalog instance.
Set up the Resources Needed for this Tutorial
In this setup, you create a policy to allow you to perform all networking operations in any compartment in your tenancy.
Perform the following steps:
A Virtual Cloud Network (VCN) is a virtual, private network that you set up in a single Oracle Cloud Infrastructure region. A VCN has a single, contiguous IPv4 CIDR block of your choice.
The allowable VCN size range is /16 to /30. Decide on the CIDR block before you create a VCN. You can't change the CIDR value later. For your reference, here's a CIDR Calculator.
To create a VCN, complete the following steps:
- Open the navigation menu , select Networking, and then select Virtual cloud networks.
- Click Create VCN.
- Enter a name to identify your VCN and select the compartment you have permission to work in. For example, Public_VCN_PE.
- Enter the CIDR block for the VCN. For this tutorial, you can enter 10.0.0.0/16.
- Select DNS RESOLUTION and enter a DNS label.
- Click Create VCN.
The VCN is created and the Virtual Cloud Networks Details page for the VCN is displayed.
By default, a route table, DHCP option, and security list are automatically created for the VCN.
Subnets are divisions you create in a VCN. Each subnet consists of a contiguous range of IP addresses that do not overlap with other subnets in the VCN.
Depending on, whether you want your subnet to have public IP addresses, you can create a public or private subnet.
Complete the following steps to create a public subnet:
- Click Create Subnet from the Virtual Cloud Networks Details page of the VCN you created in the previous step.
- Enter a name for the public subnet. For example, Public_Subnet_01.
- Retain the default Regional selection for SUBNET TYPE.
- Enter the CIDR block for the public subnet. For this tutorial, you can enter 10.0.0.0/28.
- Select the default ROUTE TABLE.
- Select Public Subnet for SUBNET ACCESS.
- Select Use DNS Hostnames in this Subnet for DNS RESOLUTION.
- Enter a DNS label.
- Select the default DHCP options and security lists.
- Click Create Subnet.
You create a private subnet when you don't want the resources created in the subnet to have public IP addresses. Complete the following steps to create a private subnet:
- Click Create Subnet from the Virtual Cloud Networks Details page of the VCN you created in the previous step.
- Enter a name for the private subnet. For example, Private_Subnet_01.
- Retain the default Regional selection for SUBNET TYPE.
- Enter the CIDR block for the private subnet. For this tutorial, you can enter 10.0.0.16/28.
- Select the default ROUTE TABLE.
- Select Private Subnet for SUBNET ACCESS.
- Select Use DNS Hostnames in this Subnet for DNS RESOLUTION.
- Enter a DNS label.
- Select the default DHCP options and security lists.
- Click Create Subnet.
Complete the following steps to create a DB system in Oracle database:
- Open the navigation menu, click Oracle Database, and then click Bare Metal, VM, and Exadata.
- In the page that appears, click Create DB System.
-
In the Create DB System page, in the DB
System Information section, enter the details as described in
the following table:
Field/Section Description Select a compartment Select your compartment. For example, Resources. Name your DB system Enter a name for your DB system. For example, DBSystem 202106151447. Select an availability domain Retain AD-1. Select a shape type Retain Virtual Machine. Select a shape Retain VM.Standard2.4. Configure the DB system In the section, enter the following details: - Total node count: Select 1.
- Oracle Database software edition: Select Enterprise Edition High Performance.
Choose storage management software Retain Oracle Grid Infrastructure. Configure storage Retain 256 for the Available storage (GB) field. Add SSH Keys Select Paste SSH Keys and paste the key in the box. Choose a license type Select Bring Your Own License (BYOL). Specify the network information In this section, enter the following details: - Virtual cloud network: Select the VCN that you created (Public_VCN_PE).
- Client subnet: Select the public subnet that your created (Public_Subnet_01).
- Hostname prefix: Enter orcl
- Host domain name: Displays the domain name based on the VCN and public subnet details. (publicsubnet01.publicvcnpe.oraclevcn.com).
- Host and domain URL: Displays the host and domain URL.
- Private IP address: This field is optional. Enter a private IP address within the CIDR block (10.0.0.0/28).
- Click Next.
-
In the Database Information section, enter the details
as described in the following table:
Field/Section Description Database name Enter a name for the database. Database unique name suffix This is an optional field. Database unique name Displays the unique name of the database. Database image Retain Oracle Database 19c. PDB name This is an optional field. Create administrator credentials In this section, enter a password to create the DB credentials. - Click Create DB System.
When you create a VCN, a security list is created by default for the VCN. You can add more security rules to this default security list or create a security list to permit traffic in and out of your VCN. In this tutorial, you add security rules to the default security list.
Depending on the subnet that you created, you create security rules for the subnet.
Complete the following steps to create a security list with the required security rules:
- Open the navigation menu , select Networking, and then select Virtual cloud networks.
- Click the VCN that you created, to view the VCN details.
- Click Security Lists from the Virtual Cloud Networks Details page of the VCN that you created.
- Click Default Security List for Public_VCN_PE.
- Click Egress Rules and do the following:
- Click Add Egress Rules.
- Enter the CIDR of your public subnet. For this tutorial, enter 10.0.0.0/16.
- Select All Protocols for IP PROTOCOL.
- Click Add Egress Rules.
- Click Ingress Rules and do the following:
- Click Add Ingress Rules.
- Enter the CIDR of your public subnet. For this tutorial, enter 10.0.0.0/16.
- Select TCP for IP PROTOCOL.
- Enter 1521–1522 for DESTINATION PORT RANGE
- Click Add Ingress Rules.
The ingress rules are added in the security list of the public subnet.
Complete the following steps to create a security rule that allows traffic from the private subnet to the public subnet:
- Open the navigation menu , select Networking, and then select Virtual cloud networks.
- Click the VCN that you created, to view the VCN details.
- Click Security Lists from the Virtual Cloud Networks Details page of the VCN that you created.
- Click the Default Security List for Public_VCN_PE.
- Click Egress Rules and do the following:
- Click Add Egress Rules.
- Enter the CIDR of your private subnet. For this tutorial, enter 10.0.0.16/28.
- Select All Protocols for IP PROTOCOL.
- Click Add Egress Rules.
- Click Ingress Rules and do the following:
- Click Add Ingress Rules.
- Enter the CIDR of your private subnet. For this tutorial, enter 10.0.0.16/28.
- Select TCP for IP PROTOCOL.
- Enter 1521–1522 for DESTINATION PORT RANGE.
- Click Add Ingress Rules.
The ingress rules are added in the security list of the public subnet.
1. Create Access Policies
To configure Data Catalog to access the private network of a data source, you need access to networking and data catalog resources.
If you already have access to perform all Data Catalog and Networking operations in your required compartments, you can skip this step.
To create the policy needed to configure a private network in data catalog, perform the following steps:
2. Obtain Data Source Details
You need the private network and database connection information for the Oracle database that you want to harvest.
Obtain the following details for the on-premise Oracle database from your administrator:
- For configuring the private network, you need the VCN and subnet name and the URL for the Oracle database.
- For creating the data asset, you need the Oracle database host, port, and database service name or SID.
- For adding a connection, you need the database login credentials.
3. Create a Private Endpoint
You create a Data Catalog private endpoint to configure the network access details for the Oracle database data sources you want to harvest.
- Open the navigation menu and select Analytics & AI. Under Data Lake, select Data Catalog.
- Click Private Endpoints.
- In the Private Endpoints page, click Create Private Endpoint.
- In the Create Private Endpoint panel that appears, do the following:
- For Create In Compartment, select the compartment in which you want to create the private endpoint. Ensure that you have permission to work in the compartment that you selected.
- In the NAME field, enter a name for the private endpoint. For example, PE_Public_Subnet.
- In the Configuration section, enter the following details:
- For Choose a VCN, select the VCN that you created. (Public_VCN_PE)
- For Subnet, select the public subnet that you created. (Public_Subnet_01)
- In the EXTERNAL DNS ZONES TO RESOLVE, enter the public subnet DNS. You can copy the DNS domain name from the details page of the public subnet that you created. (publicsubnet01.publicvcnpe.oraclevcn.com)Note
You can add multiple external DNS zones as comma-separated values.
- Click Create.
The private subnet must be in the same VCN as the Oracle database system.
- Open the navigation menu and select Analytics & AI. Under Data Lake, select Data Catalog.
- Click Private Endpoints.
- In the Private Endpoints page, click Create Private Endpoint.
- In the Create Private Endpoint panel that appears, do the following:
- For Create In Compartment, select the compartment in which you want to create the private endpoint. Ensure that you have permission to work in the compartment that you selected.
- In the NAME field, enter a name for the private endpoint. For example, PE_Public_Subnet.
- In the Configuration section, enter the following details:
- For Choose a VCN, select the VCN that you created. (Public_VCN_PE)
- For Subnet, select the public subnet that you created. (Private_Subnet_01)
- In the EXTERNAL DNS ZONES TO RESOLVE, enter the public subnet DNS. You can copy the DNS domain name from the details page of the public subnet that you created. (publicsubnet01.publicvcnpe.oraclevcn.com)Note
You can use the public subnet DNS for creating a private endpoint using private subnet. You can also add multiple external DNS zones as comma-separated values.
- Click Create.
4. Attach a Private Endpoint
You attach a private endpoint to a data catalog to allow data assets to be created for data sources available in the private network.
To attach a private endpoint to a data catalog, perform the following steps:
- Click Data Catalogs.
- Click the Actions menu for the data catalog where you want to attach the private endpoint and select Attach Private Endpoint.
- Select the private endpoint you created in the previous step and click Attach.
5. Create a Data Asset
You are now ready to register your Oracle Database system data source with Data Catalog as a data asset.
You can create a data asset by specifying the Oracle database private IP or the Oracle database FQDN.
- Click the data catalog instance to which you attached the private endpoint.
- In the Data Catalog Home tab, from the Quick Actions tile, click Create Data Asset.
- In the Create Data Asset panel that appears, do the following:
- In the Name field, enter a name for the data asset.
- In the Description field, enter a description.
- For the Type field, select Oracle Database.
- In the Host field, enter the private IP from the node of the DB system that you created. (10.0.0.3)
- In the Port field, enter 1521.
- In the Database field, enter the service name of the database system.
- Select the Use private endpoint check box.
- Click Create.
- Click the data catalog instance to which you attached the private endpoint.
- In the Data Catalog Home tab, from the Quick Actions tile, click Create Data Asset.
- In the Create Data Asset panel that appears, do the following:
- In the Name field, enter a name for the data asset.
- In the Description field, enter a description.
- For the Type field, select Oracle Database.
- In the Host field, enter the FQDN of the node of the DB system that you created.
- In the Port field, enter 1521.
- In the Database field, enter the service name of the database system.
- Select the Use private endpoint check box.
- Click Create.
After creating the data asset, you add a connection for the data asset.
For Oracle database data source types, you can use secrets in Oracle Cloud Infrastructure Vault to store the password that you need to connect to the source using a connection. By using OCI Vault, you provide the OCID of the secret when specifying the connection details, so you don't have to enter the actual password when you create the data asset.
A vault is a container for keys and secrets. Secrets store credentials such as required passwords for connecting to data sources. You use an encryption key in a vault to encrypt and import secret contents to the vault. Secret contents are based64-encoded. Data Catalog uses the same key to retrieve and decrypt secrets while connecting a data asset to the data source. For more information about vault, key, and secret, see Overview of Vault. For information about copying the secret OCID, see View Secret Details.
To add a connection for the Oracle Database data asset, follow these steps:
7. Harvest the Data Asset
You're now ready to harvest your Oracle Database data asset.
To harvest your Oracle Database data asset, perform the following steps:
What's Next
Now, you can explore the harvested data entities and attributes, create a business glossary, and link terms and tags to your data objects.