Oracle Cloud Infrastructure Documentation

Managing Encryption Keys on External Devices

Learn how to store and manage database encryption keys.

There are two options to store and manage database encryption keys for your databases on Oracle Exadata Database Service on Cloud@Customer:

  1. In an auto-login wallet file stored in an Oracle Advanced Cluster File System (Oracle ACFS) accessible by the customer VM operating system.
  2. Oracle Key Vault.

Parent topic: How-to Guides

Customer-Managed Keys in Exadata Database Service on Cloud@Customer

Customer-managed keys for Exadata Database Service on Cloud@Customer is a feature that enables you to migrate the Oracle Database TDE Master Encryption Key for an Oracle database from the password-protected wallet file stored on the Exadata Database Service on Cloud@Customer equipment to an OKV server that you control.

The Oracle Key Vault (OKV) provides fault-tolerant, highly available and scalable key and secrets management for your encrypted ExaDB-C@C databases. Use customer-managed keys when you need security governance, regulatory compliance, and homogenous encryption of data, while centrally managing, storing, and monitoring the life cycle of the keys you use to protect your data.

You can:

  • Switch from Oracle-managed keys to customer-managed keys on databases that are not enabled with Oracle Data Guard.
  • Rotate your keys to maintain security compliance.
  • Rotating the PDB key is also supported. Rotate CDB and PDB key operations are allowed only if the database is customer-managed.

Requirements

About Oracle Key Vault

Oracle Key Vault is a full-stack, security-hardened software appliance built to centralize the management of keys and security objects within the enterprise.

Note

The Oracle Key Vault is a customer-provisioned and managed system and it is not part of Oracle Cloud Infrastructure managed services.

Overview of Key Store

Integrate your on-premises Oracle Key Vault (OKV) with customer-managed database cloud services to secure your critical data on-premises.

Oracle Key Vault integration enables you to take complete control of your encryption keys and store them securely on an external, centralized key management device.

OKV is optimized for Oracle wallets, Java keystores, and Oracle Advanced Security Transparent Data Encryption (TDE) master keys. Oracle Key Vault supports the OASIS KMIP standard. The full-stack, security-hardened software appliance uses Oracle Linux and Oracle Database technology for security, availability, and scalability, and can be deployed on your choice of compatible hardware.

OKV also provides a REST interface for clients to auto-enroll endpoints and setup wallets and keys. For Autonomous Databases on Exadata Cloud@Customer to connect to OKV REST interface, create a key store in your tenancy to store the IP address and administrator credentials of your OKV. Exadata Cloud@Customer temporarily stores the OKV REST user administrator password required to connect to the OKV appliance in a password-protected wallet file so that the software running in the customer VM can connect to the OKV server. Following the migration of the TDE keys to OKV, the cloud automation software will remove the password from the wallet file. Ensure that you create a secret with Oracle's Vault Service, which will store the password required for autonomous databases to connect to OKV for key management.

For more information, see "Oracle Key Vault".

Required IAM Policy for Managing OKV on Oracle Exadata Database Service on Cloud@Customer

Review the identity access management (IAM) policy for managing OKV on Oracle Exadata Database Service on Cloud@Customer Systems.

A policy is an IAM document that specifies who has what type of access to your resources. It is used in different ways: to mean an individual statement written in the policy language; to mean a collection of statements in a single, named "policy" document (which has an Oracle Cloud ID (OCID) assigned to it), and to mean the overall body of policies your organization uses to control access to resources.

A compartment is a collection of related resources that can be accessed only by certain groups that have been given permission by an administrator in your organization.

To use Oracle Cloud Infrastructure, you must be given the required type of access in a policy written by an administrator, whether you're using the Console, or the REST API with a software development kit (SDK), a command-line interface (CLI), or some other tool. If you try to perform an action, and receive a message that you don’t have permission, or are unauthorized, then confirm with your administrator the type of access you've been granted, and which compartment you should work in.

For administrators: The policy in "Let database admins manage DB systems" lets the specified group do everything with databases and related database resources.

If you're new to policies, then see "Getting Started with Policies" and "Common Policies". If you want to dig deeper into writing policies for databases, then see "Details for the Database Service".

Tagging Resources

You can apply tags to your resources to help you organize them according to your business needs.

You can apply tags at the time you create a resource, or you can update the resource later with the desired tags. For general information about applying tags, see "Resource Tags".

Moving Resources to a Different Compartment

You can move vaults from one compartment to another.

After you move a vault to a new compartment, inherent policies apply immediately and affect access to the vault. Moving a vault doesn't affect access to any keys or secrets that the vault contains. You can move a key or secret from one compartment to another independently of moving the vault it's associated with. For more information, see "Managing Compartments".

Setting Up Your Exadata Cloud@Customer to Work With Oracle Key Vault

Prerequisites
  1. Ensure that OKV is set up and the network is accessible from the Exadata client network. Open ports 443, 5695, and 5696 for egress on the client network to access the OKV server.
  2. Ensure that the REST interface is enabled from the OKV user interface.
  3. Create "OKV REST Administrator" user.

    You can use any qualified username of your choice, for example, "okv_rest_user". For ADB-C@C and ExaDB-C@C, use the same or different REST users. Those databases can be key-managed in the same or different on-prem OKV clusters. ExaDB-C@C needs REST user with create endpoint privilege. ADB-C@C needs REST user with create endpoint and create endpoint group privileges.

  4. Gather OKV administrator credentials and IP address, which is required to connect to OKV.
  5. Open the ports 443, 5695, and 5696 for egress on the client network to access the OKV server.

For more information, see Network Port Requirements, Managing Oracle Key Vault Users, and Managing Administrative Roles and User Privileges

Step 1: Create a Vault in OCI Vault Service and Add a Secret to the Vault to Store OKV REST Administrator Password

Your Exadata Cloud@Customer infrastructure communicates with OKV over REST each time an Oracle Database is provisioned to register the Oracle Database and request a wallet on OKV. Therefore, Exadata infrastructure needs access to the REST admin credentials to register with the OKV server.

These credentials are stored securely in the Oracle Vault Service in OCI as a Secret and accessed by your Exadata Cloud@Customer infrastructure only when needed. When needed, the credentials are stored in a password-protected wallet file.

To store the OKV administrator password in the OCI Vault service, create a vault by following the instructions outlined in Managing Vaults and create a Secret in that vault by following the instructions outlined in Managing Secrets.

Step 2: Create a Dynamic Group and a Policy Statement for Key Store to Access Secret in OCI Vault

To grant your Key Store resources permission to access Secret in OCI Vault, you create an IAM dynamic group that identifies these resources and then create an IAM policy that grants this dynamic group access to the Secret you created in the OCI Vaults and Secrets.

When defining the dynamic group, you identify your Key Store resources by specifying the OCID of the compartment containing your Key Store.

  1. Copy the OCID of the compartment containing your Key Store resource.

    You can find this OCID on the Compartment Details page of the compartment.

  2. Create a dynamic group by following the instructions in "To create a dynamic group" in Oracle Cloud Infrastructure Documentation. When following these instructions, enter a matching rule of this format:
    ALL {resource.compartment.id ='<compartment-ocid>'}

    where <compartment-ocid> is the OCID of the compartment containing your Key Store resource.

  3. After creating the dynamic group, navigate to (or create) an IAM policy in a compartment higher up in your compartment hierarchy than the compartment containing your vaults and secrets. Then, add a policy statement of this format:
    allow dynamic-group <dynamic-group> to use secret-family in compartment <vaults-and-secrets-compartment>

    where <dynamic-group> is the name of the dynamic group you created and <vaults-and-secrets-compartment> is the name of the compartment in which you created your vaults and secrets.

Step 3: Create a Dynamic Group and a Policy Statement for Exadata Infrastructure to Key Store

To grant your Exadata infrastructure resources permission to access Key Store, you create an IAM dynamic group that identifies these resources and then create an IAM policy that grants this dynamic group access to the Key Store you created.

When defining the dynamic group, you identify your Exadata infrastructure resources by specifying the OCID of the compartment containing your Exadata infrastructure.

  1. Copy the OCID of the compartment containing your Exadata infrastructure resource.
    You can find this OCID on the Compartment Details page of the compartment.
  2. Create a dynamic group by following the instructions in "To create a dynamic group" in Oracle Cloud Infrastructure Documentation. When following these instructions, enter a matching rule of this format:
    ALL {resource.compartment.id ='<compartment-ocid>'}

    where <compartment-ocid> is the OCID of the compartment containing your Exadata infrastructure resource.

  3. After creating the dynamic group, navigate to (or create) an IAM policy in a compartment higher up in your compartment hierarchy than the compartment containing your Key Store. Then, add a policy statement of this format:
    Allow dynamic-group <dynamic-group> to use keystores in compartment <key-store-compartment>

    where <dynamic-group> is the name of the dynamic group you created and <key-store-compartment> is the name of the compartment in which you created your Key Store.

Step 4: Create a Policy Statement for Database Service to Use Secret from OCI Vault Service

To grant the Autonomous Database service permission to use the secret in OCI Vault to log in to the OKV REST interface, navigate to (or create) an IAM policy in a compartment higher up in your compartment hierarchy than the compartment containing your OCI Vaults and Secrets. Then, add a policy statement of this format:
allow service database to read secret-family in compartment <vaults-and-secrets-compartment>

where <vaults-and-secrets-compartment> is the name of the compartment in which you created your OCI Vaults and Secrets.

Once the OCI Vault is set up and the IAM configuration is in place, you are now ready to deploy your Oracle Key Vault 'Key Store' in OCI and associate it with your Exadata Cloud@Customer VM Cluster.

Step 5: Create Key Store

Follow these steps to create a Key Store to connect to an on-premises encryption key appliance such as Oracle Key Vault (OKV).

  1. Open the navigation menu. Under Oracle Database, click Exadata Cloud@Customer.
  2. Choose your Compartment.
  3. Click Key Stores.

    Key Stores page displays the list name of key stores, the number of databases associated with each database, and the date on which each key store was created.

  4. Click Create Key Store.
  5. In the Create Key Store dialog, enter the following general information:
    • Name your key store: A user-friendly description or other information that helps you easily identify the Key Store resource. Avoid entering confidential information.
    • Oracle Key Vault connection settings
      • Connection IP addresses: Enter at least one OKV cluster node IP address; multiple comma-separated IP addresses (of the same OKV cluster) are possible, for example, 193.10.20.1, 193.10.20.2.
      • Administrator username: Enter the user name of the okv_rest_user.
      • Administrator Password Secret: The administrator password is stored with the secret management service within OCI. Select the OCI Vault in your tenancy that contains okv_rest_user password stored as Secret.
    • Tags: Optionally, you can apply tags. If you have permission to create a resource, you also have permission to apply free-form tags to that resource. To apply a defined tag, you must have permission to use the tag namespace. For more information about tagging, see Resource Tags. If you are not sure if you should apply tags, skip this option (you can apply tags later) or ask your administrator. Avoid entering confidential information.
  6. Click Create Key Store.
  7. Ensure that you use the same "okv_rest_user" user credentials, while provisioning Autonomous Database.

    For more information, see Managing Vaults, Managing Keys, and Managing Secrets.

Managing Your Key Store

Parent topic: Managing Encryption Keys on External Devices

View Key Store Details

Follow these steps to view Key Store details that include Oracle Key Vault (OKV) connection details and the list of associated databases.

  1. Open the navigation menu. Under Oracle Database, click Exadata Cloud@Customer.
  2. Choose your Compartment.
  3. Click Key Stores.

    Key Stores page displays the list name of Key Stores, the number of databases associated with each database, and the date on which each Key Store was created.

  4. Click the name of the Key Store or click the Actions icon (three dots), and then click View Details.
  5. Click the link in the Administrator Password Secret field to view secret details.

Parent topic: Managing Your Key Store

Edit Key Store Details

You can edit a Key Store only if it is not associated with any CDBs.

  1. Open the navigation menu. Under Oracle Database, click Exadata Cloud@Customer.
  2. Choose your Compartment.
  3. Click Key Stores.
  4. Click the name of the Key Store or click the Actions icon (three dots), and then click View Details.
  5. On the Key Store Details page, click Edit.
  6. On the Edit Key Store page, make changes as needed, and then click Save Changes.

Parent topic: Managing Your Key Store

Move a Key Store to Another Compartment

Follow these steps to move a Key Store on an Oracle Exadata Cloud@Customer system from one compartment to another compartment.

  1. Open the navigation menu. Under Oracle Database, click Exadata Cloud@Customer.
  2. Choose your Compartment.
  3. Click Key Stores.
  4. Click the name of the Key Store or click the Actions icon (three dots), and then click View Details.
  5. On the Key Store Details page, click Move Resource.
  6. On the Move Resource to a Different Compartment page, select the new compartment.
  7. Click Move Resource.

Parent topic: Managing Your Key Store

Delete a Key Store

You can delete a Key Store only if it is not associated with any CDBs.

  1. Open the navigation menu. Under Oracle Database, click Exadata Cloud@Customer.
  2. Choose your Compartment.
  3. Click Key Stores.
  4. Click the name of the Key Store or click the Actions icon (three dots), and then click View Details.
  5. On the Key Store Details page, click Delete.
  6. On the Delete Key Store dialog, click Delete.

Parent topic: Managing Your Key Store

View Key Store Associated Container Database Details

Follow these steps to view details of the container database associated with a Key Store.

  1. Open the navigation menu. Under Oracle Database, click Exadata Database Service on Cloud@Customer.
  2. Choose your Compartment.
  3. Click Key Stores.
  4. In the resulting Key Stores page, click the name of the Key Store or click the Actions icon (three dots), and then click View Details.
  5. Click the name of the associated database or click the Actions icon (three dots), and then click View Details.

Parent topic: Managing Your Key Store

Using the API to Manage Key Store

Learn how to use the API to manage key store.

For information about using the API and signing requests, see "REST APIs" and "Security Credentials". For information about SDKs, see "Software Development Kits and Command Line Interface".

The following table lists the REST API endpoints to manage key store.

Operation REST API Endpoint

Create OKV Key Store

CreateKeyStore

View OKV Key Store

GetKeyStore

Update OKV Key Store

UpdateKeyStore

Delete OKV Key Store

DeleteKeyStore

Change Key store compartment

ChangeKeyStoreCompartment

Choose between customer-managed and Oracle-managed encryption

CreateAutonomousContainerDatabase

Get the Key Store (OKV or Oracle-managed) and OKV wallet name

GetAutonomousContainerDatabase

Rotate OKV and Oracle-managed key

RotateAutonomousContainerDatabaseKey

Get the Key store (OKV or Oracle-managed) and OKV wallet name

GetAutonomousDatabase

Rotate OKV and Oracle-managed key

RotateAutonomousDatabaseKey

Get the Key Store (OKV or Oracle-managed) and OKV wallet name

GetAutonomousDatabaseBackup

Administer Transparent Data Encryption (TDE) Keys

Use this procedure to change the encryption management configuration or rotate the TDE key.

After you provision a database in an ExaDB-C@C system, you can change the encryption key management to OKV and rotate the TDE key for that database.

Note

  • Oracle supports administering encryption keys on databases after and including Oracle Database 11g release 2 (11.2.0.4).
  • You can change encryption key management from Oracle-managed keys to customer-managed keys but you cannot change from customer-managed keys to Oracle-managed keys.
  • When you change to customer-managed keys on OKV, the database will experience a shutdown abort operation followed by a restart. Plan to perform the migration to customer-managed keys on OKV in a planned maintenance window.
  • To ensure that your Exadata database uses the most current version of the TDE key, rotate the key from the database details page on the Oracle Cloud Infrastructure Console. Do not use the Vault service.
  • You can rotate TDE keys only on databases that are configured with customer-managed keys.
  • You cannot rotate an encryption key:
    • when a database restore is in progress in a given Oracle Home.
    • when a database patching or database home patching is in progress.
  1. Open the navigation menu. Click Oracle Database, then click Exadata Database Service on Cloud@Customer.
  2. Choose your compartment from the Compartment drop-down.
  3. Navigate to the VM Cluster that contains the database for which you want to change encryption management or rotate a key.
    1. Under Exadata Database Service on Cloud@Customer Infrastructure, click Exadata VM Clusters.
    2. In the list of VM clusters, locate the VM cluster you want to access and click its highlighted name to view the details page for the cluster.
  4. In the Databases section, click the name of the database for which you want to change encryption management or rotate a key to display its details page.
  5. Click the More Actions drop-down.
  6. Click Administer Encryption Key.

    To change key management type from Oracle-managed keys to customer-managed keys:

    1. Click Change Key Management Type.
    2. Select Use customer-managed keys.

      You must have a valid encryption key in the Oracle Key Vault service and provide the information in the subsequent steps. For more information, see Key and Secret Management Concepts.

    3. Choose a vault from the Vault in the Compartment drop-down.
      You can change the compartment by clicking the Change Compartment link.
    4. Select an encryption key from the Master encryption key in the Compartment drop-down.
      You can change the compartment containing the encryption key you want to use by clicking the Change Compartment link.
    5. If you want to use an encryption key that you import into your vault, then select Change Compartment and enter the OCID of the key you want to use in the Key version OCID field.
    6. Click Apply.

    To rotate an encryption key on a database using customer-managed keys:

    1. Click Rotate Encryption Key to display a confirmation dialog.
    2. Click Rotate Key.
Note

  • Migration of TDE keys to Oracle Key Vault (OKV) requires 10 minutes of downtime. During the migration, the database state will be UPDATING and connections may fail due to multiple database restarts to enable OKV. Applications can resume operation after the migration completes and when the database returns to its original ACTIVE state.
  • The OKV keystore password will be set to the TDE wallet password.

Caution:

After changing key management to customer-managed keys, deleting the key from the OKV will cause the database to become unavailable.

On the database details page for this database, the Encryption section displays the encryption key name and the encryption key OCID.

How to Manually Clone a Pluggable Database (PDB) from a Remote Container Database (CDB) When Data is Encrypted with Master Encryption Key (MEK) in Oracle Key Vault (OKV)

The dbaascli tool lets you clone PDBs when the source CDB and target CDB are the same (local clone) or if they are different (remote clone). However, you cannot clone a remote PDB if the data is encrypted with a MEK in OKV.

Note

To decrypt / encrypt the data during a remote clone, the container database must have access to MEK. The MEK must be made available to the target CDB when it is stored in the OKV server.

Source CDB and Target CDB are Encrypted with MEK in the Same OKV Server

  1. Get the OKV object ID of the source PDB.
    1. Get the latest encryption key of the source PDB using SQL*Plus.
      [root@testserver oracle]# su oracle 
[oracle@testserver oracle]$ source ~/<source_db_name>.env    
[oracle@testserver oracle]$ sqlplus / as sysdba
 
SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 12 23:13:12 2023
Version 19.19.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
 
SQL> set heading off;
SQL> alter session set container=<SOURCE_PDB>;
 
Session altered.
 
SQL> select key_id,keystore_type,activation_time from v$encryption_keys order by activation_time;
 
0648E5D8D5559B4F0EBFB8AA5EE730401A
SOFTWARE KEYSTORE
25-MAR-23 12.01.41.075932 AM +00:00
 
06AFF5B6E27A954F6EBFFC77296B27C9EC
SOFTWARE KEYSTORE
25-MAR-23 11.42.51.336955 AM +00:00
 
SQL> exit
Disconnected from Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@testserver oracle]$
    2. Get the OKV object ID (uuid) of the newest MEK obtained from the step above.

      Enter the OKV Endpoint password when prompted and hit the Enter key on your keyboard. 

      [root@testserver oracle]# su oracle 
[oracle@testserver oracle]$ source ~/<source_db_name>.env 
[oracle@testserver oracle]$ $OKV_HOME/bin/okvutil list | grep 06AFF5B6E27A954F6EBFFC77296B27C9EC
E5344379-8B16-4FE9-BF35-F8ECB057571A    Symmetric Key    TDE Master Encryption Key: MKID 06AFF5B6E27A954F6EBFFC77296B27C9EC
[oracle@testserver oracle]$
  2. Install OKV REST wallet in the source database.
    1. Create the okv_rest_cli directory if it does not exist.
      [root@testserver newdb1]# su oracle
[oracle@testserver oracle]$ mkdir /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli
    2. Download and extract okvrestclipackage.zip.

      Select ALL if prompted for replacement. 

      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ cd /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli
[oracle@scaqar06dv0101 okv_rest_cli]$ curl -O -k https://<source_okv_server_ip1>:5695/okvrestclipackage.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3784k  100 3784k    0     0  19.0M      0 --:--:-- --:--:-- --:--:-- 19.1M
[oracle@testserver okv_rest_cli]$ unzip -q okvrestclipackage.zip
[oracle@testserver okv_rest_cli]$
    3. Modify the okvrestcli.ini and okvrestcli_logging.properties files as follows.
      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ vi /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini
[oracle@testserver okv_rest_cli]$ cat /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini
[Default]
server=<source_okv_server_ip1>
user=<source_okv_rest_user>
client_wallet=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet
log_property=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties
okv_client_config=/u02/app/oracle/admin/<source_db_name>/okv_home/conf/okvclient.ora
 
[oracle@testserver okv_rest_cli]$ vi /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties
[oracle@testserver okv_rest_cli]$ cat /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties  
handlers=java.util.logging.FileHandler
java.util.logging.FileHandler.pattern=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/logs/okvrest.log
java.util.logging.FileHandler.limit=200000
java.util.logging.FileHandler.count=1
java.util.logging.FileHandler.formatter=com.oracle.okv.rest.log.OkvFormatter
java.util.logging.ConsoleHandler.level=FINER
java.util.logging.ConsoleHandler.formatter=com.oracle.okv.rest.log.OkvFormatter
[oracle@testserver okv_rest_cli]$
    4. Create the client_wallet directory.
      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ mkdir /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet
[oracle@testserver okv_rest_cli]$
    5. Create OKV REST wallet using the OKV REST command-line interface.

      Enter the source OKV REST password when prompted.

      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv admin client-wallet add --client-wallet /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet --wallet-user <source_okv_rest_user>
Password:
{
  "result" : "Success"
}
[oracle@testserver okv_rest_cli]$ ls -ltr /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet
total 8
-rw------- 1 oracle oinstall    0 Jun 16 01:29 ewallet.p12.lck
-rw------- 1 oracle oinstall    0 Jun 16 01:29 cwallet.sso.lck
-rw------- 1 oracle oinstall  976 Jun 16 01:29 ewallet.p12
-rw------- 1 oracle oinstall 1021 Jun 16 01:29 cwallet.sso
[oracle@testserver okv_rest_cli]$
  3. Create a new OKV wallet to store only the PDB MEK obtained in step #1.
    1. Get the OKV wallet name from the source PDB in the format EXA_DB_NAME_DBID_PDB_NAME_WL.

      For example, the wallet name would be EXA_NEWDB1_37508325141_PDB_NAME_WL. 

      [root@testserver newdb1]# su oracle
[oracle@testserver newdb1]$ source ~/<source_db_name>.env
[oracle@testserver newdb1]$ sqlplus / as sysdba 
 
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jun 20 21:26:54 2023
Version 19.19.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
 
SQL> select name,db_unique_name,dbid from v$database; 
 
NAME      DB_UNIQUE_NAME               DBID
--------- ------------------------------ ----------
NEWDB1      newdb1_uniq             3750832514
 
SQL> select value from v$parameter where name='instance_name';
 
VALUE
--------------------------------------------------------------------------------
newdb11
 
SQL> exit
Disconnected from Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@testserver newdb1]$
    2. Create a new wallet using the OKV REST command-line interface.
      [root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet create --wallet <SOURCE_PDB_OKV_WALLET> --description "Wallet to clone <source_pdb_name> pdb from <source_db_name>" --unique FALSE
{
  "result" : "Success",
  "value" : {
    "status" : "PENDING",
    "locatorID" : "BA5FBFE1-DB41-4425-8EE4-D58541A1E41A"
  }
}
[root@testserver oracle]#
    3. Check the status until it is ACTIVE.
      [root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet check-status --wallet <SOURCE_PDB_OKV_WALLET>
{
  "result" : "Success",
  "value" : {
    "status" : "PENDING"
  }
}
[root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet check-status --wallet <SOURCE_PDB_OKV_WALLET>
{
  "result" : "Success",
  "value" : {
    "status" : "ACTIVE",
    "wallet" : "<SOURCE_PDB_OKV_WALLET>"
  }
}
[root@testserver oracle]#
  4. Add Read and Modify, and Manage Wallet permissions from the source database OKV Endpoints to the OKV wallet created in step #3.
    1. Get the Endpoint names from the source database. One per VM.

      Usually, the structure is in the format, EXA_DB_UNIQUE_NAME_DBID_SID_EP.

      For example, the Endpoint name of node 1 would be EXA_NEWDB1_UNIQ_3750832514_NEWDB11_EP. 

      [root@testserver newdb1]# su oracle
[oracle@testserver newdb1]$ source ~/<source_db_name>.env
[oracle@testserver newdb1]$ sqlplus / as sysdba 
 
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jun 20 21:26:54 2023
Version 19.19.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
 
SQL> select name,db_unique_name,dbid from v$database; 
 
NAME      DB_UNIQUE_NAME               DBID
--------- ------------------------------ ----------
NEWDB1      newdb1_uniq             3750832514
 
SQL> select value from v$parameter where name='instance_name';
 
VALUE
--------------------------------------------------------------------------------
newdb11
 
SQL> exit
Disconnected from Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@testserver newdb1]$
    2. Add Read and Modify, and Manage Wallet permissions using the OKV REST command-line interface.
      [root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet add-access --wallet <SOURCE_PDB_OKV_WALLET> --endpoint <SOURCE_OKV_EP1> --access RM_MW
{
  "result" : "Success"
}
[root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet add-access --wallet <SOURCE_PDB_OKV_WALLET> --endpoint <SOURCE_OKV_EP2> --access RM_MW
{
  "result" : "Success"
}
[root@testserver oracle]#
  5. Store MEK from the source PDB obtained in step #1 into the OKV wallet created in step #3.
    1. Add MEK (uuid obtained in step #1.b) using the OKV REST command-line interface.

      Enter the source OKV Endpoint password when prompted.

      [root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv managed-object wallet add-member --uuid E5344379-8B16-4FE9-BF35-F8ECB057571A --wallet <SOURCE_PDB_OKV_WALLET>
Password: 
{
  "result" : "Success"
}
[root@testserver oracle]#
  6. Install OKV REST wallet in the target database.
    1. Create the okv_rest_cli directory if it does not exist. 
      [root@testserver newdb1]# su oracle
[oracle@testserver oracle]$ mkdir /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli
    2. Download and extract okvrestclipackage.zip.

      Select ALL when prompted for replacement. 

      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ cd /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli
[oracle@scaqar06dv0101 okv_rest_cli]$ curl -O -k https://<target_okv_server_ip1>:5695/okvrestclipackage.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3784k  100 3784k    0     0  19.0M      0 --:--:-- --:--:-- --:--:-- 19.1M
[oracle@testserver okv_rest_cli]$ unzip -q okvrestclipackage.zip
[oracle@testserver okv_rest_cli]$
    3. Modify the okvrestcli.ini and okvrestcli_logging.properties files as follows.
      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ vi /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli.ini
[oracle@testserver okv_rest_cli]$ cat /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli.ini
[Default]
server=<target_okv_server_ip1>
user=<target_okv_rest_user>
client_wallet=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/client_wallet
log_property=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties
okv_client_config=/u02/app/oracle/admin/<target_db_name>/okv_home/conf/okvclient.ora
 
[oracle@testserver okv_rest_cli]$ vi /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties
[oracle@testserver okv_rest_cli]$ cat /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties
handlers=java.util.logging.FileHandler
java.util.logging.FileHandler.pattern=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/logs/okvrest.log
java.util.logging.FileHandler.limit=200000
java.util.logging.FileHandler.count=1
java.util.logging.FileHandler.formatter=com.oracle.okv.rest.log.OkvFormatter
java.util.logging.ConsoleHandler.level=FINER
java.util.logging.ConsoleHandler.formatter=com.oracle.okv.rest.log.OkvFormatter
[oracle@testserver okv_rest_cli]$
    4. Create the client_wallet directory.
      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ mkdir /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/client_wallet
[oracle@testserver okv_rest_cli]$
    5. Create OKV REST wallet using the OKV REST command-line interface.

      Enter the target OKV REST password when prompted.

      [oracle@testserver okv_rest_cli]$ export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/bin/okv admin client-wallet add --client-wallet /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/client_wallet --wallet-user <target_okv_rest_user>
Password:
{
  "result" : "Success"
}
[oracle@testserver okv_rest_cli]$ ls -ltr /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/client_wallet
total 8
-rw------- 1 oracle oinstall    0 Jun 16 01:29 ewallet.p12.lck
-rw------- 1 oracle oinstall    0 Jun 16 01:29 cwallet.sso.lck
-rw------- 1 oracle oinstall  976 Jun 16 01:29 ewallet.p12
-rw------- 1 oracle oinstall 1021 Jun 16 01:29 cwallet.sso
[oracle@testserver okv_rest_cli]$
  7. Add Read Only and Manage Wallet permissions from the target database OKV Endpoints to the source PDB OKV wallet created in step #3.
    1. Get the Endpoint names from the target database. One per VM.

      Usually, the structure is in the format, EXA_DB_UNIQUE_NAME_DBID_SID_EP.

      For example, the Endpoint name of node 1 would be EXA_NEWDB1_UNIQ_3750832514_NEWDB11_EP. 

      [root@testserver newdb1]# su oracle
[oracle@testserver newdb1]$ source ~/<target_db_name>.env
[oracle@testserver newdb1]$ sqlplus / as sysdba 
 
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jun 20 21:26:54 2023
Version 19.19.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
 
SQL> select name,db_unique_name,dbid from v$database; 
 
NAME      DB_UNIQUE_NAME               DBID
--------- ------------------------------ ----------
NEWDB1      newdb1_uniq             3750832514
 
SQL> select value from v$parameter where name='instance_name';
 
VALUE
--------------------------------------------------------------------------------
newdb11
 
SQL> exit
Disconnected from Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@testserver newdb1]$
    2. Add permissions Read Only and Manage Wallet using the OKV REST command-line interface.
      [root@testserver oracle]#  export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/bin/okv manage-access wallet add-access --wallet <SOURCE_PDB_OKV_WALLET> --endpoint <TARGET_OKV_EP1> --access RO_MW
{
  "result" : "Success"
}
[root@testserver oracle]#  export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/bin/okv manage-access wallet add-access --wallet <SOURCE_PDB_OKV_WALLET> --endpoint <TARGET_OKV_EP2> --access RO_MW
{
  "result" : "Success"
}
[root@testserver oracle]#
  8. Clone the PDB.
    1. Run dbaascli to clone the PDB.

      Enter the source DB SYS user password when prompted.

      [root@testserver oracle]# dbaascli pdb remoteClone --pdbName <source_pdb_name> --dbName <target_db_name> --sourceDBConnectionString <source_db_connection_string> --targetPDBName <target_pdb_name>
DBAAS CLI version 23.2.1.0.0
Executing command pdb remoteClone --pdbName <source_pdb_name> --dbName <target_pdb_name> --sourceDBConnectionString scaqar06dvclu01-scan1.us.oracle.com:1521/<source_db_unique_name>.us.oracle.com --targetPDBName <target_pdb_name>
Job id: 197f30e9-209e-4ec5-9700-a13f7915f8b9
Session log: /var/opt/oracle/log/alyokv1/pdb/remoteClone/dbaastools_2023-06-12_10-32-17-PM_188384.log
Enter REMOTE_DB_SYS_PASSWORD:
 
Enter REMOTE_DB_SYS_PASSWORD (reconfirmation):
 
Loading PILOT...
Session ID of the current execution is: 6848
Log file location: /var/opt/oracle/log/alyokv1/pdb/remoteClone/pilot_2023-06-12_10-32-35-PM_204184
-----------------
Running Plugin_initialization job
Enter REMOTE_DB_SYS_PASSWORD
***************
Completed Plugin_initialization job
-----------------
Running Validate_input_params job
Completed Validate_input_params job
-----------------
Running Perform_dbca_prechecks job
Completed Perform_dbca_prechecks job
-----------------
Running PDB_creation job
Completed PDB_creation job
-----------------
Running Load_pdb_details job
Completed Load_pdb_details job
-----------------
Running Configure_pdb_service job
Completed Configure_pdb_service job
-----------------
Running Configure_tnsnames_ora job
Completed Configure_tnsnames_ora job
-----------------
Running Set_pdb_admin_user_profile job
Completed Set_pdb_admin_user_profile job
-----------------
Running Lock_pdb_admin_user job
Completed Lock_pdb_admin_user job
-----------------
Running Register_ocids job
Skipping. Job is detected as not applicable.
-----------------
Running Prepare_blob_for_standby_in_primary job
Skipping. Job is detected as not applicable.
-----------------
Running Generate_dbsystem_details job
Completed Generate_dbsystem_details job
dbaascli execution completed
[root@testserver oracle]#
  9. Delete the source PDB OKV wallet created in step #3 using the OKV REST command-line interface.
    [root@testserver oracle]#  export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet delete --wallet  <SOURCE_PDB_OKV_WALLET>
{
  "result" : "Success"
}
[root@testserver oracle]#
  10. Delete the OKV REST wallet created in step #2.
    1. Delete the wallet files in the dbaas_acfs directory.
      [root@testserver oracle]# rm -f /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet/*
[root@testserver oracle]#
  11. Delete OKV REST wallet created in step #6.
    1. Delete the wallet files in the dbaas_acfs directory.
      [root@testserver oracle]# rm -f /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/client_wallet/*
[root@testserver oracle]#

Source CDB and Target CDB are Encrypted with MEK in a Different OKV Server

  1. Get the OKV object ID of the source PDB.
    1. Get the latest encryption key of the source PDB using SQL*Plus.
      [root@testserver oracle]# su oracle 
[oracle@testserver oracle]$ source ~/<source_db_name>.env    
[oracle@testserver oracle]$ sqlplus / as sysdba
 
SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 12 23:13:12 2023
Version 19.19.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
 
SQL> set heading off;
SQL> alter session set container=<SOURCE_PDB>;
 
Session altered.
 
SQL> select key_id,keystore_type,activation_time from v$encryption_keys order by activation_time;
 
0648E5D8D5559B4F0EBFB8AA5EE730401A
SOFTWARE KEYSTORE
25-MAR-23 12.01.41.075932 AM +00:00
 
06AFF5B6E27A954F6EBFFC77296B27C9EC
SOFTWARE KEYSTORE
25-MAR-23 11.42.51.336955 AM +00:00
 
SQL> exit
Disconnected from Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@testserver oracle]$
    2. Get the OKV object ID (uuid) of the newest MEK obtained from the step above.

      Enter the OKV Endpoint password when prompted and hit the Enter key on your keyboard. 

      [root@testserver oracle]# su oracle 
[oracle@testserver oracle]$ source ~/<source_db_name>.env 
[oracle@testserver oracle]$ $OKV_HOME/bin/okvutil list | grep 06AFF5B6E27A954F6EBFFC77296B27C9EC
E5344379-8B16-4FE9-BF35-F8ECB057571A    Symmetric Key    TDE Master Encryption Key: MKID 06AFF5B6E27A954F6EBFFC77296B27C9EC
[oracle@testserver oracle]$
  2. Install OKV REST wallet in the source database.
    1. Create the okv_rest_cli directory if it does not exist.
      [root@testserver newdb1]# su oracle
[oracle@testserver oracle]$ mkdir /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli
    2. Download and extract okvrestclipackage.zip.

      Select ALL if prompted for replacement. 

      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ cd /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli
[oracle@scaqar06dv0101 okv_rest_cli]$ curl -O -k https://<source_okv_server_ip1>:5695/okvrestclipackage.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3784k  100 3784k    0     0  19.0M      0 --:--:-- --:--:-- --:--:-- 19.1M
[oracle@testserver okv_rest_cli]$ unzip -q okvrestclipackage.zip
[oracle@testserver okv_rest_cli]$
    3. Modify the okvrestcli.ini and okvrestcli_logging.properties files as follows.
      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ vi /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini
[oracle@testserver okv_rest_cli]$ cat /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini
[Default]
server=<source_okv_server_ip1>
user=<source_okv_rest_user>
client_wallet=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet
log_property=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties
okv_client_config=/u02/app/oracle/admin/<source_db_name>/okv_home/conf/okvclient.ora
 

[oracle@testserver okv_rest_cli]$ vi /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties
[oracle@testserver okv_rest_cli]$ cat /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties  
handlers=java.util.logging.FileHandler
java.util.logging.FileHandler.pattern=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/logs/okvrest.log
java.util.logging.FileHandler.limit=200000
java.util.logging.FileHandler.count=1
java.util.logging.FileHandler.formatter=com.oracle.okv.rest.log.OkvFormatter
java.util.logging.ConsoleHandler.level=FINER
java.util.logging.ConsoleHandler.formatter=com.oracle.okv.rest.log.OkvFormatter
[oracle@testserver okv_rest_cli]$
    4. Create the client_wallet directory.
      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ mkdir /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet
[oracle@testserver okv_rest_cli]$
    5. Create OKV REST wallet using OKV REST command-line interface.

      Enter the source OKV REST password when prompted.

      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv admin client-wallet add --client-wallet /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet --wallet-user <source_okv_rest_user>
Password:
{
  "result" : "Success"
}
[oracle@testserver okv_rest_cli]$ ls -ltr /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet
total 8
-rw------- 1 oracle oinstall    0 Jun 16 01:29 ewallet.p12.lck
-rw------- 1 oracle oinstall    0 Jun 16 01:29 cwallet.sso.lck
-rw------- 1 oracle oinstall  976 Jun 16 01:29 ewallet.p12
-rw------- 1 oracle oinstall 1021 Jun 16 01:29 cwallet.sso
[oracle@testserver okv_rest_cli]$
  3. Create a new OKV wallet to store only the PDB MEK obtained in step #1.
    1. Create a new wallet using the OKV REST command-line interface. Name suggested: EXA_DB_NAME_DBID_PDB_NAME_WL
      [root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet create --wallet <SOURCE_PDB_OKV_WALLET> --description "Wallet to clone <source_pdb_name> pdb from <source_db_name>" --unique FALSE
{
  "result" : "Success",
  "value" : {
    "status" : "PENDING",
    "locatorID" : "BA5FBFE1-DB41-4425-8EE4-D58541A1E41A"
  }
}
[root@testserver oracle]#
    2. Check the status until it is ACTIVE.
      [root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet check-status --wallet <SOURCE_PDB_OKV_WALLET>
{
  "result" : "Success",
  "value" : {
    "status" : "PENDING"
  }
}
[root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet check-status --wallet <SOURCE_PDB_OKV_WALLET>
{
  "result" : "Success",
  "value" : {
    "status" : "ACTIVE",
    "wallet" : "<SOURCE_PDB_OKV_WALLET>"
  }
}
[root@testserver oracle]#
  4. Add Read and Modify, and Manage Wallet permissions from the source database OKV Endpoints to the OKV wallet created in step #3.
    1. Get the Endpoint names from the source database. One per VM.

      Usually, the structure is in the format, EXA_DB_UNIQUE_NAME_DBID_SID_EP.

      For example, the Endpoint name of node 1 would be EXA_NEWDB1_UNIQ_3750832514_NEWDB11_EP. 

      [root@testserver newdb1]# su oracle
[oracle@testserver newdb1]$ source ~/<source_db_name>.env
[oracle@testserver newdb1]$ sqlplus / as sysdba 
 
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jun 20 21:26:54 2023
Version 19.19.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
 
SQL> select name,db_unique_name,dbid from v$database; 
 
NAME      DB_UNIQUE_NAME               DBID
--------- ------------------------------ ----------
NEWDB1      newdb1_uniq             3750832514
 
SQL> select value from v$parameter where name='instance_name';
 
VALUE
--------------------------------------------------------------------------------
newdb11
 
SQL> exit
Disconnected from Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@testserver newdb1]$
    2. Add permissions Read and Modify, and Manage Wallet using the OKV REST command-line interface.
      [root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet add-access --wallet <SOURCE_PDB_OKV_WALLET> --endpoint <SOURCE_OKV_EP1> --access RM_MW
{
  "result" : "Success"
}
[root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet add-access --wallet <SOURCE_PDB_OKV_WALLET> --endpoint <SOURCE_OKV_EP2> --access RM_MW
{
  "result" : "Success"
}
[root@testserver oracle]#
  5. Store MEK from the source PDB obtained in step #1 into OKV wallet created in step #3.
    1. Add MEK (uuid obtained in step #1.b) using the OKV REST command-line interface.

      Enter the source OKV Endpoint password when prompted.

      [root@testserver oracle]# export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv managed-object wallet add-member --uuid E5344379-8B16-4FE9-BF35-F8ECB057571A --wallet <SOURCE_PDB_OKV_WALLET>
Password: 
{
  "result" : "Success"
}
[root@testserver oracle]#
  6. Download OKV wallet created in step #3 from the OKV server to the local filesystem.
    1. Create a new directory with permissions for oracle user.

      This directory will store the wallet that will contain only the MEK of the source PDB.

      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ mkdir /home/oracle/<source_pdb_wallet_dir>
[oracle@testserver oracle]$
    2. Download the OKV wallet created in step #3 to the directory created in step #6.a using okvutil.

      It will prompt twice for a password to encrypt the local wallet. Use the same password as the source Endpoint password. Also, enter the source Endpoint password when prompted.

      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ source ~/nfsa.env
[oracle@testserver oracle]$ $OKV_HOME/bin/okvutil download -l /home/oracle/<source_pdb_wallet_dir> -t wallet -g <SOURCE_PDB_OKV_WALLET>
Enter new wallet password (<enter> for auto-login): 
Confirm new wallet password: 
Enter Oracle Key Vault endpoint password: 
Download succeeded
[oracle@testserver oracle]$
    3. Zip the wallet directory.
      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ cd /home/oracle
[oracle@testserver oracle]$ zip -r <source_pdb_wallet_dir>.zip <source_pdb_wallet_dir>  
  adding: <source_pdb_wallet_dir>/ (stored 0%)
  adding: <source_pdb_wallet_dir>/ewallet.p12 (stored 0%)
[oracle@testserver oracle]$
  7. Delete the source PDB OKV wallet created in step #3.
    [root@testserver oracle]#  export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/bin/okv manage-access wallet delete --wallet  <SOURCE_PDB_OKV_WALLET>
{
  "result" : "Success"
}
[root@testserver oracle]#
  8. Delete the OKV REST wallet created in step #1.
    1. Delete the wallet files in the dbaas_acfs directory.
      [root@testserver oracle]# rm -f /var/opt/oracle/dbaas_acfs/<source_db_name>/okv_rest_cli/client_wallet/*
[root@testserver oracle]#
  9. Copy the source PDB wallet downloaded to the focal filesystem in step #6 to the target Cluster VM.
  10. Delete the source PDB wallet from the source local filesystem created in step #6.
    1. Delete the wallet directory.
      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ rm -rf /home/oracle/<source_pdb_wallet_dir>
[oracle@testserver oracle]$
    2. Delete the wallet zip file. 
      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ rm -f /home/oracle/<source_pdb_wallet_dir>.zip
[oracle@testserver oracle]$
  11. Install OKV REST wallet in the target database.
    1. Create the okv_rest_cli directory if it does not exist. 
      [root@testserver newdb1]# su oracle
[oracle@testserver oracle]$ mkdir /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli
    2. Download and extract okvrestclipackage.zip.

      Select ALL when prompted for replacement. 

      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ cd /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli
[oracle@testserver okv_rest_cli]$ curl -O -k https://<target_okv_server_ip>:5695/okvrestclipackage.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3784k  100 3784k    0     0  19.0M      0 --:--:-- --:--:-- --:--:-- 19.1M
[oracle@testserver okv_rest_cli]$ unzip -q okvrestclipackage.zip
[oracle@testserver okv_rest_cli]$
    3. Modify the okvrestcli.ini and okvrestcli_logging.properties files as follows.
      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ vi /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli.ini
[oracle@testserver okv_rest_cli]$ cat /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli.ini
[Default]
server=<target_okv_server_ip1>
user=<target_okv_rest_user>
client_wallet=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/client_wallet
log_property=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties
okv_client_config=/u02/app/oracle/admin/<target_db_name>/okv_home/conf/okvclient.ora
 
[oracle@testserver okv_rest_cli]$ vi /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties
[oracle@testserver okv_rest_cli]$ cat /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli_logging.properties  
handlers=java.util.logging.FileHandler
java.util.logging.FileHandler.pattern=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/logs/okvrest.log
java.util.logging.FileHandler.limit=200000
java.util.logging.FileHandler.count=1
java.util.logging.FileHandler.formatter=com.oracle.okv.rest.log.OkvFormatter
java.util.logging.ConsoleHandler.level=FINER
java.util.logging.ConsoleHandler.formatter=com.oracle.okv.rest.log.OkvFormatter
[oracle@testserver okv_rest_cli]$
    4. Create the client_wallet directory.
      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ mkdir /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/client_wallet
[oracle@testserver okv_rest_cli]$
    5. Create OKV REST wallet using the OKV REST command-line interface.

      Enter the target OKV REST password when prompted.

      [root@testserver oracle]# su oracle
[oracle@testserver okv_rest_cli]$ export JAVA_HOME=/usr/java/latest; export OKV_RESTCLI_CONFIG=/var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/conf/okvrestcli.ini; /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/bin/okv admin client-wallet add --client-wallet /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/client_wallet --wallet-user <target_okv_rest_user>
/var/opt/oracle/dbaas_acfs/newdb1/okv_rest_cli/logs/okvrest.log.lck
Password:
{
  "result" : "Success"
}
[oracle@testserver okv_rest_cli]$ ls -ltr /var/opt/oracle/dbaas_acfs/newdb1/okv_rest_cli/client_wallet
total 8
-rw------- 1 oracle oinstall    0 Jun 16 01:29 ewallet.p12.lck
-rw------- 1 oracle oinstall    0 Jun 16 01:29 cwallet.sso.lck
-rw------- 1 oracle oinstall  976 Jun 16 01:29 ewallet.p12
-rw------- 1 oracle oinstall 1021 Jun 16 01:29 cwallet.sso
[oracle@testserver okv_rest_cli]$
  12. Upload the source PDB wallet created in step #6 and copied to the target Cluster VM in step #9.
    1. Unzip the source PDB wallet.
      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ cd /home/oracle/
[oracle@testserver ~]$ unzip <source_pdb_wallet_dir>.zip 
Archive:  nfsa_1672104454_NFSPDB_wallet.zip
   creating: <source_pdb_wallet_dir>/
 extracting: <source_pdb_wallet_dir>/ewallet.p12  
[oracle@testserver ~]$
    2. Get OKV wallet name from the target database in the format EXA_DB_NAME_DBID_WL.

      For example, the wallet name would be EXA_NEWDB1_37508325141_WL. 

      [root@testserver newdb1]# su oracle
[oracle@testserver newdb1]$ source ~/<target_db_name>.env
[oracle@testserver newdb1]$ sqlplus / as sysdba 
 
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jun 20 21:26:54 2023
Version 19.19.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
 
SQL> select name,db_unique_name,dbid from v$database; 
 
NAME      DB_UNIQUE_NAME               DBID
--------- ------------------------------ ----------
NEWDB1      newdb1_uniq             3750832514
 
SQL> select value from v$parameter where name='instance_name';
 
VALUE
--------------------------------------------------------------------------------
newdb11
 
SQL> exit
Disconnected from Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@testserver newdb1]$
    3. Upload the source PDB wallet to the target OKV wallet using okvutil.

      Enter the source PDB wallet password when prompted. Use the same password as the source Endpoint password.

      Also, enter the target Endpoint password when prompted.

      [root@testserver oracle]# su oracle
[oracle@testserver oracle]$ source ~/<target_db_name>.env
[oracle@testserver oracle]$ $OKV_HOME/bin/okvutil upload -t WALLET -l /home/oracle/<source_pdb_wallet_dir> -g <TARGET_OKV_WALLET>
Enter source wallet password: 
Enter Oracle Key Vault endpoint password: 
WARNING: Object ORACLE.SECURITY.ID.ENCRYPTION. already exists; use -o to overwrite
Upload succeeded
[oracle@testserver oracle]$
  13. Clone the PDB.
    1. Run dbaascli to clone the PDB.
      [root@testserver oracle]# dbaascli pdb remoteClone --pdbName <source_pdb_name> --dbName <target_db_name> --sourceDBConnectionString <source_db_connection_string> --targetPDBName <target_pdb_name> 
DBAAS CLI version 23.2.1.0.0
Executing command pdb remoteClone --pdbName <source_pdb_name> --dbName <target_db_name> --sourceDBConnectionString scaqar06dvclu01-scan1.us.oracle.com:1521/<source_db_unique_name>.us.oracle.com --targetPDBName <target_pdb_name>
Job id: 7d4f638a-1f3a-4219-a05a-0215588dcae8
Session log: /var/opt/oracle/log/alyokv1/pdb/remoteClone/dbaastools_2023-06-13_01-29-09-AM_179996.log
Enter REMOTE_DB_SYS_PASSWORD:
 
Enter REMOTE_DB_SYS_PASSWORD (reconfirmation):
 
Loading PILOT...
Session ID of the current execution is: 6857
Log file location: /var/opt/oracle/log/alyokv1/pdb/remoteClone/pilot_2023-06-13_01-29-21-AM_196991
-----------------
Running Plugin_initialization job
Enter REMOTE_DB_SYS_PASSWORD
*************
Completed Plugin_initialization job
-----------------
Running Validate_input_params job
Completed Validate_input_params job
-----------------
Running Perform_dbca_prechecks job
Completed Perform_dbca_prechecks job
-----------------
Running PDB_creation job
Completed PDB_creation job
-----------------
Running Load_pdb_details job
Completed Load_pdb_details job
-----------------
Running Configure_pdb_service job
Completed Configure_pdb_service job
-----------------
Running Configure_tnsnames_ora job
Completed Configure_tnsnames_ora job
-----------------
Running Set_pdb_admin_user_profile job
Completed Set_pdb_admin_user_profile job
-----------------
Running Lock_pdb_admin_user job
Completed Lock_pdb_admin_user job
-----------------
Running Register_ocids job
Skipping. Job is detected as not applicable.
-----------------
Running Prepare_blob_for_standby_in_primary job
Skipping. Job is detected as not applicable.
-----------------
Running Generate_dbsystem_details job
Completed Generate_dbsystem_details job
dbaascli execution completed
[root@testserver oracle]#
  14. Delete the OKV REST wallet created in step #1.
    1. Delete the wallet files in the dbaas_acfs directory.
      [root@testserver oracle]# rm -f /var/opt/oracle/dbaas_acfs/<target_db_name>/okv_rest_cli/client_wallet/*
[root@testserver oracle]#