Known Issues

Details: The default Oracle Cloud Infrastructure Resolver resolves public URL endpoints (and URL endpoints with public hostnames) to IP addresses. Additionally, a subnet can be configured with a custom DNS server that resolves internal hostname (and private hostname) URL endpoints to IP addresses. However, API gateways you create with the API Gateway service do not inherit custom DNS servers from subnets. Instead API gateways use the default Oracle Cloud Infrastructure Resolver, which does not resolve internal/private hostname URL endpoints.

Due to this restriction, if you create an API gateway that has an internal/private hostname URL endpoint as the HTTP or HTTPS URL back end, calls to the API will fail because the hostname cannot be resolved to an IP address.

Workaround: We're working on a resolution. In the meantime, if you want to create an API gateway that has an internal/private URL endpoint as the HTTP or HTTPS URL back end, you must specify the host's IP address in the URL rather than the hostname. In addition, if the back end is an HTTPS URL, you must also select the Disable SSL Verification option in the Console (or include isSSLVerifyDisabled: true in the API deployment specification JSON file).

Direct link to this issue: API gateways do not inherit custom DNS servers from subnets

Details: In Synthetic Monitoring, the Browser and Scripted Browser monitors might fail to run against applications that use frames.

Workaround: We are aware of the issue and working on a resolution. For Scripted Browser monitors, you can work around this issue by replacing index=<frame-index> with either id=<id-of-frame> or name=<name-of-frame> in the .side script.

For example, if this script is the original version:

{
      "id": "23956f51-8812-40e6-ac91-1d608871ee4c",
      "comment": "",
      "command": "selectFrame",
      "target": "index=0",
      "targets": [
        ["index=0"]
      ],
      "value": ""
    }

The following script would be the modified version:

{
      "id": "23956f51-8812-40e6-ac91-1d608871ee4c",
      "comment": "",
      "command": "selectFrame",
      "target": "id=frame1",
      "targets": [
        ["id=frame1"]
      ],
      "value": ""
    }

Direct link to this issue: Browser and Scripted Browser monitors might not run applications that use frames

Details: Authorization policies based on the apm-domains resource tags do not work for the Trace Explorer and Synthetic Monitoring APIs, causing authorization failures.

Workaround: We are aware of the issue and working on a resolution.

Direct link to this issue: Issues with the authorization policies based on the apm-domains resource tags

Details: When you try to enable cross-region replication for a volume configured to use a Vault encryption key, the following error message occurs: Edit Volume Error: You cannot enable cross-region replication for volume <volume_ID> as it uses a Vault encryption key.

Workaround: We're working on a resolution. Cross-region replication is not supported for volumes encrypted with a customer-managed key. As a workaround to enable replication, unassign the Vault encryption key from the volume. In this scenario, the volume is encrypted with an Oracle-managed key.

Direct link to this issue: Cross-region replication not supported for volumes encrypted with customer-managed keys

Details: To achieve the optimal performance level for volumes configured for ultra high performance, the volume attachment must be multipath-enabled. Multipath-enabled attachments to VM instances are only supported for instances based on shapes with 16 or greater OCPUs.

If you have an instance with fewer than 16 OCPUs, you can resize it so that it has 16 or more OCPUs to support multipath-enabled attachments. This step will not work for instances where the original number of OCPUs was less than 8 and the volume attachment is paravirtualized. In this scenario, after the volume is detached and reattached, the volume attachment will still not be multipath-enabled even though the instance now supports multipath-enabled attachments.

Workaround: As a workaround, we recommend that you create a new instance based on a shape with 16 or more OCPUs, and then attach the volume to the new instance.

Direct link to this issue: Paravirtualized volume attachment not multipath-enabled after instance is resized

Details: When you attempt to attach the maximum number of block volumes to a smaller VM.Standard.A1.Flex instance, in some cases, the volumes might fail to attach. This happens because of limitations with the underlying physical host configuration.

Workaround: We're working on a resolution. As a workaround, we recommend that you increase the size of the VM by resizing the VM, and then try attaching the volumes again.

Direct link to this issue: Attaching the maximum number of block volumes to smaller VM.Standard.A1.Flex instances might fail

Details: The copy_boot_volume_backup_and_wait_for_state and copy_volume_backup_and_wait_for_state from the BlockStorage Client Composite operations throw a 404/NotAuthorizedOrNotFound when copying a backup from one region to another. For more information see: https://github.com/oracle/oci-python-sdk/issues/344.

Workaround: Instead of using the composite operations, use two different clients for this operation; one client in the Source Region to send the request for copying the backup from Region A to Region B, and a second client in Destination region to wait for the backup to become available. See example here: https://github.com/oracle/oci-python-sdk/blob/master/examples/copy_volume_backup_example.py

Direct link to this issue: Python SDK composite operations throw a 404 NotAuthorizedOrNotFound error even though operation is a success

Details: When you schedule volume and volume group backups using a backup policy that is enabled for cross-region copy for volumes that are encrypted using Vault service encryption keys, the encryption keys are not copied with the volume backup to the destination region. The volume backup copies in the destination region are instead encrypted using Oracle-provided keys.

Workaround: We're working on a resolution. As a workaround, you can manually copy volume backups and volume group backups across regions, either manually or using a script, and specify the key management key ID in the target region for the copy operation. For more information about manual cross region copy, see Copying a Volume Backup Between Regions.

Direct link to this issue: Vault encryption keys not copied to destination region for scheduled cross region backup copies

Details: When you attach a Windows boot volume as a data volume to another instance, when you try to connect to the volume using the steps described in Connecting to a Volume the volume fails to attach and you may encounter the following error:

Connect-IscsiTarget : The target has already been logged in via an iSCSI session.

Workaround: You need to append the following to the Connect-IscsiTarget command copied from the Console:

-IsMultipathEnabled $True

Direct link to this issue: Attaching a Windows boot volume as a data volume to another instance fails

Details: When calling GetInstance, the bootVolumeSizeInGBs attribute of InstanceSourceViaImageDetails is null.

Workaround: We're working on a resolution. To work around this issue, call GetBootVolume, and use the sizeInGBs attribute of BootVolume.

Direct link to this issue: bootVolumeSizeInGBs attribute is null

Details: Due to some issues with one of its dependencies, the Go SDK feature which allows customers to automatically use new realms which might be unknown to the SDK is not functioning from within Cloud Shell.

can not create client, bad configuration: failed to get security token: failed to renew security token: failed to get security token: failed to call: Post "https://<endpoint>/v1/x509": dial tcp: lookup <endpoint> on 127.0.0.11:53: server misbehaving
panicked while retrying operation. Panic was: runtime error: invalid memory address or nil pointer dereference

Workaround: To resolve this issue, enable resolving regions using the instance metadata service for Go SDK. For more information, see: Adding Regions

Direct link to this issue: Go SDK cannot automatically find some regions while running in Cloud Shell

Details: When you try to access the Console using Firefox, the Console page never loads in the browser. This problem is likely caused by a corrupted Firefox user profile.

Workaround: Create a new Firefox user profile as follows:

1. Ensure that you are on the latest version of Firefox. If not, update to the latest version.
2. Create a new user profile and remove your old user profile. See Mozilla Support for instructions to create and remove user profiles: https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles.
3. Open Firefox with the new profile.

Alternatively, you can use one of the other Supported Browsers.

Direct link to this issue: Bug in the Firefox browser can cause the Console not to load

Details: The properties of new worker nodes starting in a node pool do not reflect the latest changes to the node pool's properties. The likely cause is use of the deprecated quantityPerSubnet and subnetIds attributes when using the UpdateNodePoolDetails API operation to update node pool properties.

Direct link to this issue: Worker node properties out-of-sync with updated node pool properties

Details: When you launch the Kubernetes Dashboard, in some situations you might encounter "net/http: TLS handshake timeout" and "connection reset by peer" error messages in your web browser. This issue has only been observed in newly created clusters running Kubernetes version 1.11. For details about a related Kubernetes issue, see https://github.com/kubernetes/dashboard/issues/3038.

Workaround:

1. In a terminal window, enter:

   $ kubectl -n kube-system port-forward svc/kubernetes-dashboard 8443:443

2. In your web browser, go to https://localhost:8443

Direct link to this issue: Unable to launch Kubernetes Dashboard

Details: When you use a Kubeconfig token version 2.0.0 to access Helm/Tiller versions prior to version 2.11, you will receive one of the following errors:

• Error: Unauthorized
• Error: could not get Kubernetes client: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1beta1"

Workaround: Upgrade Helm/Tiller as follows:

1. In a terminal window, download a Kubeconfig token version 1.0.0 by entering the following command:

   $ oci ce cluster create-kubeconfig --token-version=1.0.0 --cluster-id=<cluster_ocid>

2. Identify the region key to use to specify the Oracle Cloud Infrastructure Registry registry in the cluster's region (see Availability by Region). For example, if the cluster is in US East (Ashburn), iad is the region key to use to specify the registry in that region.

3. Upgrade Tiller by entering the following command:

   $ helm init --upgrade -i <region-key>.ocir.io/odx-oke/oke-public/tiller:v2.14.3

   where <region-key> is the key that you identified in the previous step.

4. In a browser, navigate to https://helm.sh/docs/using_helm/#installing-the-helm-client and follow the instructions to download and install the Helm client binary.

5. Having upgraded Helm/Tiller, download a Kubeconfig token version 2.0.0 by entering the following command:

   $ oci ce cluster create-kubeconfig --token-version=2.0.0 --cluster-id=<cluster_ocid>

Direct link to this issue: Unable to access in-cluster Helm

Details: The Container Engine for Kubernetes 1.8.0 release included a security improvement to improve cipher strength on the kubelet running on customer worker nodes. New worker nodes created between August 20, 2019 and September 16, 2019 include this configuration. The new set of ciphers does not allow connections to the kubelet via http/2. This restriction impacts the Metric Server, and also the Horizontal Pod Autoscaler which depends on the Metrics Server.

Workaround:

For each existing worker node in turn:

1. Prevent new pods from starting and delete existing pods on the worker node by entering kubectl drain <node_name>. For more information:

   • about using kubectl, see Accessing a Cluster Using Kubectl
   • about the drain command, see drain in the Kubernetes documentation

   Recommended: Leverage pod disruption budgets as appropriate for your application to ensure that there's a sufficient number of replica pods running throughout the drain operation.

2. Delete the worker node (for example, by terminating it in the Console).
3. Wait for a replacement worker node to start.

The replacement worker nodes include include new settings to enable communication with the kubelet.

Direct link to this issue: Some Kubernetes features (for example, the Metrics Server) cannot communicate with the kubelet via http/2

Details: When a new pod starts on a worker node in a cluster, in some situations the pod fails to mount all volumes attached to the node due to timeouts and you see a message similar to the following:

Unable to mount volumes for pod "<pod_name>(<pod_uid>)": timeout expired waiting for volumes to attach or mount for pod "<namespace>"/"<pod_name>". list of unmounted volumes=[<failed_volume>]. list of unattached volumes=[<… list of volumes >]

One possible cause identified for this issue is if the pod spec includes an fsGroup field in the securityContext field. If the container is running on a worker node as a non-root user, setting the fsGroup field in the securityContext can cause timeouts due to the number of files to which Kubernetes must make ownership changes (see https://github.com/kubernetes/kubernetes/issues/67014).

If the pod spec does not include an fsgroup field in the securityContext, the cause is unknown.

Workarounds:

If the pod spec includes the fsgroup field in the securityContext and the container is running a non-root user, consider the following workarounds:

• Remove the fsgroup field from the securityContext.
• Use the supplementalGroups field in the securityContext (instead of fsgroup), and set supplementalGroups to the volume identifier.
• Change the pod spec so that the container runs as root.

If the pod spec does not include the fsgroup field in the securityContext, or if the container is already running as root, you have to restart or replace the worker node. For example, by stopping and starting the instance, by rebooting the instance, or by terminating the instance so that a new instance is started. Follow the instructions in Stopping and Starting an Instance or Terminating an Instance as appropriate to use the Console or the API. Alternatively you can use CLI commands, such as the following example to terminate an instance:

$ INSTANCE_OCID=$(kubectl get node <name> -ojsonpath='{.spec.providerID}')
$ oci compute instance terminate --instance-id $INSTANCE_OCID

where <name> is the worker node name, derived from the Private IP Address property of the instance (for example, 10.0.10.5).

Direct link to this issue: Kubernetes pods fail to mount volumes due to timeouts

Details: When using the OS Management service to manage operating system updates and patches on Oracle Cloud Infrastructure instances, there are some situations in which cluster node pools created by Container Engine for Kubernetes fail to come online.

Workaround:

There are two possible workarounds:

• Workaround 1: If you want to use OS Management to manage Oracle Cloud Infrastructure instances, enable Oracle Enterprise Linux in OS Management. See Managing Software Sources.
• Workaround 2: If you don't want to use OS Management to manage Oracle Cloud Infrastructure instances, make sure there are no policies that allow OS Management to run. Specifically, remove the policy that grants a dynamic group of instances access to the OS Management service. See Setting Up Policies for OS Management.

Direct link to this issue: OS Management causes Kubernetes cluster node pools to fail

Details: If node pools have master nodes running Kubernetes version 1.19 (or later) and worker nodes running Kubernetes version 1.18 (or earlier), mounting block volumes attached to the cluster using the FlexVolume volume plugin might not work as expected. For example, you might see:

• A FailedMount warning message in the events of a pod running on a worker node, even though the block volume has been attached successfully.
• A Volume not attached according to node status for volume error message in the logs of the kubelet running on a worker node.

Workaround:

1. If there isn't already a node pool in the cluster with worker nodes running Kubernetes version 1.19 (or later), add such a node pool now.
2. Remove the affected worker node that is running Kubernetes version 1.18 (or earlier), as follows:
   1. Prevent new pods from starting and delete existing pods on the affected worker node by entering kubectl drain <node_name>. For more information:
      • about using kubectl, see Accessing a Cluster Using Kubectl
      • about the drain command, see drain in the Kubernetes documentation
   2. Delete the affected worker node (for example, by terminating it in the Console).

Direct link to this issue: Volume mount issues in node pools with master nodes running Kubernetes version 1.19 (or later) and worker nodes running Kubernetes version 1.18 (or earlier)

/ # nslookup www.oracle.com
;; reply from unexpected source: 10.244.0.167#53, expected 10.96.5.5#53
;; reply from unexpected source: 10.244.0.167#53, expected 10.96.5.5#53
;; reply from unexpected source: 10.244.0.167#53, expected 10.96.5.5#53
;; connection timed out; no servers could be reached 

sudo /usr/sbin/lsmod | grep br_netfilter 

If no results are returned, then the Bridge Netfilter kernel module is not enabled. The Bridge Netfilter kernel module is required to masquerade VxLAN traffic for Kubernetes pods.

sudo modprobe br_netfilter 
sudo sh -c 'echo "br_netfilter" > /etc/modules-load.d/br_netfilter.conf'

Direct link to this issue: Issues resolving with DNS (nslookup, dig, or curl)

Details: When using VCN-native pod networking, the source client IP address of inbound requests to a pod might not be preserved as expected. Instead, inbound requests received via a Kubernetes service of type LoadBalancer that has externalTrafficPolicy: Local set in the manifest file might be shown as originating from the worker node's IP address.

Workaround: For inbound TCP requests received via a Kubernetes service of type LoadBalancer that has the oci.oraclecloud.com/load-balancer-type: "lb" annotation in the manifest file, obtain the source client IP address from the X-Forwarded-For or X-Real-IP header.

Direct link to this issue: Source client IP is not preserved for traffic through a LoadBalancer Service using externalTrafficPolicy: Local

Details: When using VCN-native pod networking, pods might be unable to communicate over the network if you have specified a bare metal shape for worker nodes in one or more of the node pools in the cluster.

Workaround: Specify a VM shape for worker nodes in every node pool in the cluster when using VCN-native pod networking.

Direct link to this issue: Pod network connectivity issues on bare metal instances

Details: When using VCN-native pod networking, the maximum number of pods per worker node in a node pool might be limited to 31, regardless of the number of OCPUs you specify for the flexible shape you have selected for the node pool.

Workaround: If you want more than 31 pods per worker node in a node pool, select a different shape for worker nodes in the node pool.

Direct link to this issue: Incorrect maximum pods per node limit for flexible shapes

Details: When using VCN-native pod networking, pods running on worker nodes connected to a pod subnet with a CIDR block outside the first CIDR block specified for the VCN might be unable to communicate with Kubernetes services.

Workaround: Create pod subnets with CIDR blocks within the first CIDR block specified for the VCN.

Direct link to this issue: Pod network connectivity issues on VCNs with added CIDR blocks

Details: When using the Node Doctor script to troubleshoot node issues, the script might display an exception error message similar to the following:

FileNotFoundError: [Errno 2] No such file or directory: ’/home/opc/vendor/pip…

The Node Doctor script will probably continue to run and, having displayed the message Generating node doctor bundle, produce troubleshooting output.

Workaround: We are aware of the issue and working on a resolution. In the meantime, if the Node Doctor script displays the message Generating node doctor bundle, note that the troubleshooting output is still valid.

If you do not want the Node Doctor script to display the FileNotFoundError: [Errno 2]... exception error message, update the Node Doctor script by entering:

node-doctor.sh --update

For more information about the Node Doctor script and how to update it, see Node Troubleshooting Using the Node Doctor Script.

Direct link to this issue: Node Doctor script displays FileNotFoundError: [Errno 2] exception

Details: If a cluster uses VCN-native pod networking and has both a workload pod and the CoreDNS pod running on the same worker node, DNS resolution sometimes fails because traffic is incorrectly NATed.

Workaround: We are aware of the issue and working on a resolution.

In the meantime, if you encounter this issue, run the kube-proxy daemonset on worker nodes in ipvs mode (rather than in the default iptables mode).

To move workloads to worker nodes running the kube-proxy daemonset in ipvs mode, do one of the following

• Create a new nodepool using the CLI. Specify that the kube-proxy daemonset is to run in ipvs mode on new worker nodes in the new node pool, using --node-metadata '{ "oke-kubeproxy-proxy-mode": "ipvs"}'. For example:

  oci ce node-pool create \
  --cluster-id <cluster-ocid> \
  --name <node-pool-name> \
  ...
  --node-metadata '{ "oke-kubeproxy-proxy-mode": "ipvs"}' \
  ...

  Having created the new node pool, move workloads from the existing node pool to the new node pool. Drain existing worker nodes in the original node pool to prevent new pods starting and to delete existing pods. Then delete the original node pool. When new worker nodes start in the new node pool, they run the kube-proxy daemonset in ipvs mode as you specified. For more information, see Performing an Out-of-Place Worker Node Update by Replacing an Existing Node Pool with a New Node Pool.

• Update an existing node pool using the CLI. Specify that the kube-proxy daemonset is to run in ipvs mode on new worker nodes created in the existing node pool, using --node-metadata '{ "oke-kubeproxy-proxy-mode": "ipvs"}'. For example:

  oci ce node-pool update \
  --node-pool-id <node-pool-ocid> \
  ...
  --node-metadata '{ "oke-kubeproxy-proxy-mode": "ipvs"}' \
  ...

  Having updated the node pool, delete each existing worker node in turn. Select appropriate cordon and drain options to prevent new pods starting and to delete existing pods. Then start a new worker node to take the place of each worker node you delete. When new worker nodes start in the existing node pool, they run the kube-proxy daemonset in ipvs mode as you specified. For more information, see Performing an In-Place Worker Node Update by Updating an Existing Node Pool.

Direct link to this issue: DNS resolution sometimes fails in clusters using VCN-native pod networking

Details: If a cluster uses VCN-native pod networking and has worker nodes running Oracle Linux 8 (OL8), pods sometimes fail to start on one of the worker nodes. The issue has the following characteristics:

• The worker node is running an OL8 image.
• Host-network related pods run as expected on the worker node, but all other pods fail to start.
• The crictl logs contain the message Adding address to IPVLAN parent device (indicating that an IP address is being attached to the worker node's secondary VNIC), followed by the error message Error adding secondary VNIC IP.
• Running the Linux ip address command on the worker node shows that one (or more) secondary VNICs does not have an attached IP address.
• All (or most) other worker nodes are operating as expected.

A likely cause identified for this issue is related to the NetworkManager, which manages network devices and connections. In some cases, the NetworkManager detaches the IP address attached to one or more of the worker node's secondary VNICs, causing the OCI VCN-Native Pod Networking CNI plugin to fail.

Workaround: We are aware of the issue and working on a resolution.

In the meantime, if you encounter this issue, delete the affected worker node (see Deleting Worker Nodes). Assuming you delete the worker node without scaling down the node pool, the deleted worker node is replaced and the replacement worker node operates as expected.

Direct link to this issue: Pods sometimes fail to start on a worker node running Oracle Linux 8, in clusters using VCN-native pod networking

Details: Existing PDBs do not appear in a newly created database and it may take up to a few hours before they appear in the Console. This includes the default PDB for a new database and existing PDBs for cloned or restored databases. In case of an in-place restore to an older version, the PDB list is updated similarly and may have some delay.

Workaround: None

Direct link to this issue: Existing PDBs in a new database

Details: Creating and cloning a PDB in the primary database is not allowed via console or the API.

Workaround: You can use sqlplus to create or clone PDBs in the Primary database and they will be synced later in OCI console.

Direct link to this issue: PDB in existing Data Guard configuration

Details: Using the Database Service API to migrate a file-based TDE wallet to a customer-managed key-based TDE wallet on Oracle Database 12c release 1 (12.1.0.2) fails with the following error:

[FATAL] [DBAAS-11014] - Required patches (30128047) are not present in the Oracle home <ORACLE_HOME>
ACTION: Apply the required patches (30128047) and re-try the operation

nohup /var/opt/oracle/dbaascli/dbaascli tde file_to_hsm --dbname <database_name> --kms_key_ocid <kms_key_ocid> --skip_patch_check true &

In the preceding command, <kms_key_ocid> refers to the OCID of the customer-managed key you are using.

Details: Using the Database Service API to migrate a customer-managed key-based TDE wallet to a file-based TDE wallet on Oracle Database 12c release 1 (12.1.0.2) fails with the following error:

[FATAL] [DBAAS-11014] - Required patches (30128047) are not present in the Oracle home <ORACLE_HOME>
ACTION: Apply the required patches (30128047) and re-try the operation

nohup /var/opt/oracle/dbaascli/dbaascli tde hsm_to_file --dbname <database_name> --skip_patch_check true &

Details: Using the Database Service API to migrate a file-based TDE wallet to customer-managed key-based TDE wallet on Oracle Database 12c release 2 (12.2.0.1) fails with the following error:

[FATAL] [DBAAS-11014] - Required patches (30128047) are not present in the Oracle home <ORACLE_HOME>
ACTION: Apply the required patches (30128047) and re-try the operation

Workaround: Migrate a file-based TDE wallet to a customer-managed key-based TDE wallet, as follows:

1. Determine whether the database has encrypted UNDO or TEMP tablespaces in any of the Autonomous Databases or in CDB$ROOT, as follows:
   Run the following query from CDB$ROOT, to list all encrypted tablespaces contained within all Autonomous Databases:

   SQL> select tstab.con_id, tstab.name from v$tablespace tstab, v$encrypted_tablespaces enctstab where tstab.ts# = enctstab.ts# and encryptedts = 'YES';

   In then NAME column of the result of the query, search for the names of UNDO and TEMP tablespaces. If there are encrypted UNDO or TEMP tablespaces, then proceed to the next step.

2. Unencrypt UNDO or TEMP tablespaces, as follows:

   If an UNDO tablespace is encrypted

   Unencrypt existing UNDO tablespaces, as follows:

   SQL> alter tablespace <undo_tablespace_name> encryption online decrypt;

   Repeat this procedure for all encrypted UNDO tablespaces.

   If a TEMP tablespace is encrypted

   1. Check the default TEMP tablespace, as follows:

      SQL> select property_value from database_properties where property_name = 'DEFAULT_TEMP_TABLESPACE';

      If the default TEMP tablespace is not encrypted but other TEMP tablespaces are encrypted, then drop the other TEMP tablespaces, as follows:

      SQL> drop tablespace <temp_tablespace_name>;

      Skip the remainder of the steps in this procedure.

      If the default TEMP tablespace is encrypted, then proceed with the remaining steps to create and set an unencrypted default TEMP tablespace.

   2. Set the encrypt_new_tablespaces parameter to DDL, as follows:

      SQL> alter system set "encrypt_new_tablespaces" = DDL scope = memory;

   3. Create a TEMP tablespace with the specifications of the current TEMP tablespace, as follows:

      SQL> create temporary tablespace <temp_tablespace_name> TEMPFILE size 5000M;

   4. Set the new TEMP tablespace as the default TEMP tablespace for the database, as follows:

      SQL> alter database default temporary tablespace <temp_tablespace_name>;

   5. Drop existing TEMP tablespaces, as follows:

      SQL> drop tablespace <temp_tablespace_name>;

   Repeat this procedure for all encrypted TEMP tablespaces.

   The database is now running with default UNDO and TEMP tablespaces that are not encrypted and any older TEMP and UNDO tablespaces are also decrypted.

   Set encrypt_new_tablespaces to its original value, as follows:

   SQL> alter system set "encrypt_new_tablespaces" = cloud_only;

   Proceed with keystore migration to customer-managed keys.

3. Once you confirm that there are no UNDO or TEMP tablespaces encrypted in any of the pluggable databases or in CDB$ROOT, use the DBAASCLI utility with the --skip_patch_check true flag to skip the validation of the patch for bug 30128047. Ensure that you have applied the patch for bug 31527103 in the Oracle home and then run following dbaascli command:

   nohup /var/opt/oracle/dbaascli/dbaascli tde file_to_hsm --dbname <database_name> --kms_key_ocid <kms_key_ocid> --skip_patch_check true &

   In the preceding command, <kms_key_ocid> refers to the OCID of the customer-managed key you are using.

Details: Using the Database Service API to migrate a customer-managed key-based TDE wallet to a file-based TDE wallet on Oracle Database 12c release 2 (12.2.0.1) fails with the following error:

[FATAL] [DBAAS-11014] - Required patches (30128047) are not present in the Oracle home <ORACLE_HOME>
ACTION: Apply the required patches (30128047) and re-try the operation

Workaround: Migrate a customer-managed key-based TDE wallet to a file-based TDE wallet, as follows:

1. Determine whether the database has encrypted UNDO or TEMP tablespaces in any of the Autonomous Databases or in CDB$ROOT, as follows:
   Run the following query from CDB$ROOT, to list all encrypted tablespaces contained within all Autonomous Databases:

   SQL> select tstab.con_id, tstab.name from v$tablespace tstab, v$encrypted_tablespaces enctstab where tstab.ts# = enctstab.ts# and encryptedts = 'YES';

   In then NAME column of the result of the query, search for the names of UNDO and TEMP tablespaces. If there are encrypted UNDO or TEMP tablespaces, then proceed to the next step.

2. Unencrypt UNDO or TEMP tablespaces, as follows:

   If an UNDO tablespace is encrypted

   Unencrypt existing UNDO tablespaces, as follows:

   SQL> alter tablespace <undo_tablespace_name> encryption online decrypt;

   Repeat this procedure for all encrypted UNDO tablespaces.

   If a TEMP tablespace is encrypted

   1. Check the default TEMP tablespace, as follows:

      SQL> select property_value from database_properties where property_name = 'DEFAULT_TEMP_TABLESPACE';

      If the default TEMP tablespace is not encrypted but other TEMP tablespaces are encrypted, then drop the other TEMP tablespaces, as follows:

      SQL> drop tablespace <temp_tablespace_name>;

      Skip the remainder of the steps in this procedure.

      If the default TEMP tablespace is encrypted, then proceed with the remaining steps to create and set an unencrypted default TEMP tablespace.

   2. Set the encrypt_new_tablespaces parameter to DDL, as follows:

      SQL> alter system set "encrypt_new_tablespaces" = DDL scope = memory;

   3. Create a TEMP tablespace with the specifications of the current TEMP tablespace, as follows:

      SQL> create temporary tablespace <temp_tablespace_name> TEMPFILE size 5000M;

   4. Set the new TEMP tablespace as the default TEMP tablespace for the database, as follows:

      SQL> alter database default temporary tablespace <temp_tablespace_name>;

   5. Drop existing TEMP tablespaces, as follows:

      SQL> drop tablespace <temp_tablespace_name>;

   Repeat this procedure for all encrypted TEMP tablespaces.

   The database is now running with default UNDO and TEMP tablespaces that are not encrypted and any older TEMP and UNDO tablespaces are also decrypted.

   Set encrypt_new_tablespaces to its original value, as follows:

   SQL> alter system set "encrypt_new_tablespaces" = cloud_only;

   Proceed with keystore migration to customer-managed keys.

3. Once you confirm that there are no UNDO or TEMP tablespaces encrypted in any of the pluggable databases or in CDB$ROOT, use the DBAASCLI utility with the --skip_patch_check true flag to skip the validation of the patch for bug 30128047. Ensure that you have applied the patch for bug 29667994 in the Oracle home and then run following dbaascli command:

   nohup /var/opt/oracle/dbaascli/dbaascli tde file_to_hsm --dbname <database_name> --kms_key_ocid <kms_key_ocid> --skip_patch_check true &

   In the preceding command, <kms_key_ocid> refers to the OCID of the customer-managed key you are using.

Details: When you change the license type of your Database or DB system from BYOL to license included, or the other way around, you are billed for both types of licenses for the first hour. After that, you are billed according to your updated license type.

Workaround: We're working on a resolution.

Direct link to this issue: Billing issue when changing license type

Details: If you configure your VCN with a service gateway, the private subnet blocks access to the YUM repositories needed to update the OS. This issue affects all types of DB systems.

Workaround: This issue is now resolved. Here is the workaround that was recommended before the issue's resolution:

The service gateway enables access to the Oracle YUM repos if you use the Available Service CIDR Labels called All <region> Services in Oracle Services Network. However, you still might have issues accessing the YUM services through the service gateway. There's a solution to the issue. For details, see Access issues for instances to Oracle yum services through service gateway.

Direct link to this issue: Service gateway does not currently support OS updates

Details: Unmanaged backups to Object Storage using the database CLI (dbcli) or RMAN fail with the following errors:

-> Oracle Error Codes found:
-> ORA-19554: error allocating device, device type: SBT_TAPE, device name:
-> ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
-> KBHS-00712: ORA-29024 received from local HTTP service
-> ORA-27023: skgfqsbi: media manager protocol error

In response to policies implemented by two common web browsers regarding Symantec certificates, Oracle recently changed the certificate authority used for Oracle Cloud Infrastructure. The resulting change in SSL certificates can cause backups to Object Storage to fail if the Oracle Database Cloud Backup Module still points to the old certificate.

Workaround for dbcli: Check the log files for the errors listed and, if found, update the backup module.

Review the RMAN backup log files for the errors listed above:

1. Determine the ID of the failed backup job.

   dbcli list-jobs

   In this example output, the failed backup job ID is "f59d8470-6c37-49e4-a372-4788c984ea59".

   root@<node name> ~]# dbcli list-jobs
    
   ID                                       Description                                                                 Created                             Status
   ---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
   cbe852de-c0f3-4807-85e8-7523647ec78c     Authentication key update for DCS_ADMIN                                     March 30, 2018 4:10:21 AM UTC       Success
   db83fdc4-5245-4307-88a7-178f8a0efa48     Provisioning service creation                                               March 30, 2018 4:12:01 AM UTC       Success
   c1511a7a-3c2e-4e42-9520-f156b1b4cf0e     SSH keys update                                                             March 30, 2018 4:48:24 AM UTC       Success
   22adf146-9779-4a2c-8682-7fd04d7520b2     SSH key delete                                                              March 30, 2018 4:50:02 AM UTC       Success
   6f2be750-9823-4ed5-b5ff-8e49f136dd22     create object store:bV0wqIaoLA4xLT4dGjOu                                    March 30, 2018 5:33:38 AM UTC       Success
   0716f464-1a10-40df-a303-cadee0302b1b     create backup config:bV0wqIaoLA4xLT4dGjOu_BC                                March 30, 2018 5:33:49 AM UTC       Success
   e08b21c3-cd09-4e3a-944c-d1da96cb21d8     update database : hfdb1                                                     March 30, 2018 5:34:04 AM UTC       Success
   1c3d7c58-79c3-4039-8f48-787057ce7c6e     Create Longterm Backup with TAG-DBTLongterm<identity number> for Db:<dbname>    March 30, 2018 5:37:11 AM UTC       Success
   f59d8470-6c37-49e4-a372-4788c984ea59     Create Longterm Backup with TAG-DBTLongterm<identity number> for Db:<dbname>    March 30, 2018 5:43:45 AM UTC       Failure

2. Use the ID of the failed job to obtain the location of the log file to review.

   
   dbcli describe-job -i <failed_job_ID>

   Relevant output from the describe-job command should look like this:

   Message: DCS-10001:Internal error encountered: Failed to run Rman statement.
   Refer log in Node <node_name>: /opt/oracle/dcs/log/<node_name>/rman/bkup/<db_unique_name>/rman_backup/<date>/rman_backup_<date>.log.

Update the Oracle Database Cloud Backup Module:

1. Determine the Swift object store ID and user the database is using for backups.

   1. Run the dbcli list-databases command to determine the ID of the database.

   2. Use the database ID to determine the backup configuration ID (backupConfigId).

      dbcli list-databases
      dbcli describe-database -i <database_ID> -j

   3. Using the backup configuration ID you noted from the previous step, determine the object store ID (objectStoreId).

      dbcli list-backupconfigs
      dbcli describe-backupconfig –i <backupconfig_ID> -j

   4. Using the object store ID you noted from the previous step, determine the object store user (userName).

      dbcli list-objectstoreswifts
      dbcli describe-objectstoreswift –i <objectstore_ID> -j

2. Using the object store credentials you obtained from step 1, update the backup module.

   dbcli update-objectstoreswift –i <objectstore_ID> -p –u <user_name>

Workaround for RMAN: Check the RMAN log files for the error messages listed. If found, log on to the host as the oracle user, and use your Swift credentials to reinstall the backup module.

java -jar <opc_install.jar_path> -opcId '<swift_user_ID>' -opcPass '<auth_token>' -container <objectstore_container> -walletDir <wallet_directory> -configfile <config_file> -host https://swiftobjectstorage.<region_name>.oraclecloud.com/v1/<object_storage_namespace> -import-all-trustcerts

For a multi-node DB system, perform the workaround on all nodes in the cluster.

See Oracle Database Cloud Backup Module documentation for details on using this command.

Direct link to this issue: Backing up to Object Storage using dbcli or RMAN fails due to certificate change

Details: The SDKs released on October 18, 2018 introduce code-breaking changes to the database size and the database edition attributes in the database backup APIs.

Workaround: Refer to the following language-specific documentation for more details about the breaking changes, and update your existing code as applicable:

Direct link to this issue: Breaking changes in Database service SDKs

Details: Backup and restore operations might not work in your DB system when you use the Console or the API.

Workaround: Install the Oracle Database Cloud Backup Module, and then contact Oracle Support Services for further instructions.

To install the Oracle Database Cloud Backup Module:

1. SSH to the DB system, and log in as opc.

   
   ssh -i <SSH_key> opc@<DB_system_IP address>
   login as: opc

   Alternatively, you can use opc@<DB_system_hostname> to log in.

2. Download the Oracle Database Cloud Backup Module from http://www.oracle.com/technetwork/database/availability/oracle-cloud-backup-2162729.html.
3. Extract the contents of opc_installer.zip to a target directory, for example, /home/opc.
4. In your tenancy, create a temporary user, and grant them privileges to access the tenancy's Object Storage.
5. For this temporary user, create an Working with Auth Tokens and note down the password.

6. Verify that credentials work by running the following curl command:

   Note
   
   Swift passwords are now called "Auth tokens." For details, see Using an Auth Token with Swift.

   curl -v -X HEAD -u  <user_id>:'<auth_token>' https://swiftobjectstorage.<region_name>.oraclecloud.com/v1/<object_storage_namespace>

   See https://cloud.oracle.com/infrastructure/storage/object-storage/faq for the correct region to use.

   The command should return either the HTTP 200 or the HTTP 204 No Content success status response code. Any other status code indicates a problem connecting to Object Storage.

7. Run the following command:

   java -jar opc_install.jar -opcid <user_id> -opcPass '<auth_token>' -libDir <target_dir> -walletDir <target_dir> -host https://swiftobjectstorage.<region_name>.oraclecloud.com/v1/<object_storage_namespace> -configFile config.txt

   Note that <target_dir> is the directory to which you extracted opc_installer.zip in step 3.

   This command might take a few minutes to complete because it downloads libopc.so and other files. Once the command completes, you should see several files (including libopc.so) in your target directory.

8. Change directory to your target directory, and copy the lipopc.so and opc_install.jar files into the /opt/oracle/oak/pkgrepos/oss/odbcs directory.

   cp libopc.so /opt/oracle/oak/pkgrepos/oss/odbcs

   
   
   cp opc_install.jar /opt/oracle/oak/pkgrepos/oss/odbcs

   (You might have to use sudo with the copy commands to run them as root.)

9. Run the following command to check whether the directory indicated exists:

   
   
   ls /opt/oracle/dcs/commonstore/pkgrepos/oss/odbcs

   If this directory exists, perform the following steps:

   1. Back up the files in the /opt/oracle/dcs/commonstore/pkgrepos/oss/odbcs directory.

   2. Run these two commands to replace the existing libopc.so and opc_install.jar files in that directory:

      
      cp libopc.so /opt/oracle/dcs/commonstore/pkgrepos/oss/odbcs
      cp opc_install.jar /opt/oracle/dcs/commonstore/pkgrepos/oss/odbcs

10. Verify the version of opc_install.jar.

    
    java -jar /opt/oracle/oak/pkgrepos/oss/odbcs/opc_install.jar |grep -i build
    

    If /opt/oracle/dcs/commonstore/pkgrepos/oss/odbcs exists, also run the following command:

    
    java -jar /opt/oracle/dcs/commonstore/pkgrepos/oss/odbcs/opc_install.jar |grep -i build

    Both commands should return the following output:

    Oracle Database Cloud Backup Module Install Tool, build MAIN_2017-08-16.

11. (Optional) Delete the temporary user and the target directory you used to install the backup module.

After you complete the procedure, contact Oracle Support or your tenant administrator for further instructions. You must provide the OCID of the DB system for which you would like to enable backups.

Direct link to this issue: Unable to use Managed Backups in your DB System

Details: Memory limitations of host machines running the VM.Standard1.1 shape can cause failures for automatic database backup jobs managed by Oracle Cloud Infrastructure (jobs managed by using either the Console or the API). You can change the systems' memory parameters to resolve this issue.

Workaround: Change the systems' memory parameters as follows:

1. Switch to the oracle user in the operating system.

   [opc@hostname ~]$ sudo su - oracle

2. Set the environment variable to login to the database instance. For example:

   
   [oracle@hostname ~]$ . oraenv
    ORACLE_SID = [oracle] ? orcl
   				

3. Start SQL*Plus.

   [oracle@hostname ~]$ sqlplus / as sysdba

4. Change the initial memory parameters as follows:

   
   SQL> ALTER SYSTEM SET SGA_TARGET = 1228M scope=spfile;
   SQL> ALTER SYSTEM SET PGA_AGGREGATE_TARGET = 1228M;
   SQL> ALTER SYSTEM SET PGA_AGGREGATE_LIMIT = 2457M;
   SQL> exit
   							

5. Restart the database instance.

   
   [oracle@hostname ~]$ srvctl stop database -d db_unique_name -o immediate
   [oracle@hostname ~]$ srvctl start database -d db_unique_name -o open								

Direct link to this issue: Managed Automatic Backups fail on the VM.Standard1.1 shape due to a process crash

Details: On High Performance and Extreme Performance DB systems, Data Pump utility operations that use compression and/or parallelism might fail and return the error ORA-00439: feature not enabled. This issue affects database versions 12.1.0.2.161018 and 12.1.0.2.170117.

Workaround: Apply patch 25579568 or 25891266 to Oracle Database homes for database versions 12.1.0.2.161018 or 12.1.0.2.170117, respectively. Alternatively, use the Console to apply the April 2017 patch to the DB system and database home.

Direct link to this issue: Oracle Data Pump operations return "ORA-00439: feature not enabled"

Details: You might get a "Secure Connection Failed" error message when you try to connect to the EM Express console from your 1-node DB system because the correct permissions were not applied automatically.

Workaround: Add read permissions for the asmadmin group on the wallet directory of the DB system, and then retry the connection:

1. SSH to the DB system host, log in as opc, sudo to the grid user.

   [opc@dbsysHost ~]$ sudo su - grid
   [grid@dbsysHost ~]$ . oraenv
   ORACLE_SID = [+ASM1] ?
   The Oracle base has been set to /u01/app/grid
   

2. Get the location of the wallet directory, shown in red below in the command output.

   [grid@dbsysHost ~]$ lsnrctl status | grep xdb_wallet
   
   (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=dbsysHost.sub04061528182.dbsysapril6.oraclevcn.com)(PORT=5500))(Security=(my_wallet_directory=/u01/app/oracle/admin/dbsys12_phx3wm/xdb_wallet))(Presentation=HTTP)(Session=RAW))

3. Return to the opc user, switch to the oracle user, and change to the wallet directory.

   [opc@dbsysHost ~]$ sudo su - oracle
   [oracle@dbsysHost ~]$ cd /u01/app/oracle/admin/dbsys12_phx3wm/xdb_wallet

4. List the directory contents and note the permissions.

   
   [oracle@dbsysHost xdb_wallet]$ ls -ltr
   total 8
   -rw------- 1 oracle asmadmin 3881 Apr  6 16:32 ewallet.p12
   -rw------- 1 oracle asmadmin 3926 Apr  6 16:32 cwallet.sso
   

5. Change the permissions:

   
   [oracle@dbsysHost xdb_wallet]$ chmod 640 /u01/app/oracle/admin/dbsys12_phx3wm/xdb_wallet/*

6. Verify that read permissions were added.

   [oracle@dbsysHost xdb_wallet]$ ls -ltr
   total 8
   -rw-r----- 1 oracle asmadmin 3881 Apr  6 16:32 ewallet.p12
   -rw-r----- 1 oracle asmadmin 3926 Apr  6 16:32 cwallet.sso
   

Direct link to this issue: Unable to connect to the EM Express console from your 1-node DB system

Details: Backup operations to Object Storage using the Exadata backup utility (bkup_api) or RMAN fail with the following errors:

* DBaaS Error trace:
-> API::ERROR -> KBHS-00715: HTTP error occurred 'oracle-error'
-> API::ERROR -> ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
-> API::ERROR -> ORA-19554: error allocating device, device type: SBT_TAPE, device name:
-> API::ERROR -> ORA-27023: skgfqsbi: media manager protocol error
-> API::ERROR Unable to verify the backup pieces
-> Oracle Error Codes found:
-> ORA-19554: error allocating device, device type: SBT_TAPE, device name:
-> ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
-> KBHS-00712: ORA-29024 received from local HTTP service
-> ORA-27023: skgfqsbi: media manager protocol error

In response to policies implemented by two common web browsers regarding Symantec certificates, Oracle recently changed the certificate authority used for Oracle Cloud Infrastructure. The resulting change in SSL certificates can cause backups to Object Storage to fail if the Oracle Database Cloud Backup Module still points to the old certificate.

Workaround for bkup_api: Check the log files for the errors listed above, and if found, reinstall the backup module.

Use the following command to check the status of the failed backup:

/var/opt/oracle/bkup_api/bkup_api bkup_status --dbname=<database_name>

Run the following command to reinstall the backup module:

/var/opt/oracle/ocde/assistants/bkup/bkup -dbname=<database_name>

Workaround for RMAN: Check the RMAN log files for the error messages listed. If found, log on to your host as the oracle user, and reinstall the backup module using your Swift credentials.

java -jar <opc_install.jar_path> -opcId '<Swift_user_ID>' -opcPass '<auth_token>' -container <objectstore_container> -walletDir <wallet_directory> -configfile <config_file> -host https://swiftobjectstorage.<region_name>.oraclecloud.com/v1/<object_storage_namespace> -import-all-trustcerts

Perform this workaround on all nodes in the cluster.

See Oracle Database Cloud Backup Module documentation for details on using this command.

Direct link to this issue: Backing up to Object Storage using bkup_api or RMAN fails due to certificate change

Details: With the release of the shared Database Home feature for Exadata DB systems, the Console now also synchronizes and displays information about databases that are created and managed by using the dbaasapi and dbaascli utilities. However, databases with Data Guard configured do not display correct information in the Console under the following conditions:

• If Data Guard was enabled by using the Console, and then a change is made to the primary or standby database by using dbaascli (such as moving the database to a different home), the result is not reflected in the Console.
• If Data Guard was configured manually, the Console does not show a Data Guard association between the two databases.

Workaround: We are aware of the issue and working on a resolution. In the meantime, Oracle recommends that you manage your Data Guard enabled databases by using either only the Console or only command line utilities.

Direct link to this issue: Console information not synced for Data Guard enabled databases when using dbaascli

Details: This is a clusterware issue that occurs only when the Oracle GI version is 12.2.0.1 without any bundle patch. The problem is caused by corruption of a voting disk after you offline then online the disk.

Workaround: Determine the version of the GI, and whether the voting disk is corrupted. Repair the disk, if applicable, and then apply the latest GI bundle.

1. Verify the GI version is 12.2.0.1 without any bundle patch applied:

   
   [root@rmstest-udaau1 ~]# su - grid
   [grid@rmstest-udaau1 ~]$ . oraenv
   ORACLE_SID = [+ASM1] ? +ASM1
   The Oracle base has been set to /u01/app/grid
   [grid@rmstest-udaau1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
   Oracle Interim Patch Installer version 12.2.0.1.6
   Copyright (c) 2018, Oracle Corporation.  All rights reserved.
   
   
   Oracle Home       : /u01/app/12.2.0.1/grid
   Central Inventory : /u01/app/oraInventory
      from           : /u01/app/12.2.0.1/grid/oraInst.loc
   OPatch version    : 12.2.0.1.6
   OUI version       : 12.2.0.1.4
   Log file location : /u01/app/12.2.0.1/grid/cfgtoollogs/opatch/opatch2018-01-15_22-11-10PM_1.log
   
   Lsinventory Output file location : /u01/app/12.2.0.1/grid/cfgtoollogs/opatch/lsinv/lsinventory2018-01-15_22-11-10PM.txt
   
   --------------------------------------------------------------------------------
   Local Machine Information::
   Hostname: rmstest-udaau1.exaagclient.sretest.oraclevcn.com
   ARU platform id: 226
   ARU platform description:: Linux x86-64
   
   Installed Top-level Products (1):
   
   Oracle Grid Infrastructure 12c                                       12.2.0.1.0
   There are 1 products installed in this Oracle Home.
   
   
   There are no Interim patches installed in this Oracle Home.
   
   
   --------------------------------------------------------------------------------
   
   OPatch succeeded.

2. Check the /u01/app/grid/diag/crs/<hostname>/crs/trace/ocssd.trc file for evidence that the GI failed to start due to voting disk corruption:

   ocssd.trc
    
   2017-01-17 23:45:11.955 :    CSSD:3807860480: clssnmvDiskCheck:: configured 
   Sites = 1, Incative sites = 1, Mininum Sites required = 1 
   2017-01-17 23:45:11.955 :    CSSD:3807860480: (:CSSNM00018:)clssnmvDiskCheck: 
   Aborting, 2 of 5 configured voting disks available, need 3 
   ...... 
   . 
   2017-01-17 23:45:11.956 :    CSSD:3807860480: clssnmCheckForNetworkFailure: 
   skipping 31 defined 0 
   2017-01-17 23:45:11.956 :    CSSD:3807860480: clssnmRemoveNodeInTerm: node 4, 
   slcc05db08 terminated. Removing from its own member and connected bitmaps 
   2017-01-17 23:45:11.956 :    CSSD:3807860480: 
   ################################### 
   2017-01-17 23:45:11.956 :    CSSD:3807860480: clssscExit: CSSD aborting from 
   thread clssnmvDiskPingMonitorThread 
   2017-01-17 23:45:11.956 :    CSSD:3807860480: 
   ################################### 
   2017-01-17 23:45:11.956 :    CSSD:3807860480: (:CSSSC00012:)clssscExit: A 
   fatal error occurred and the CSS daemon is terminating abnormally 
    
   ------------
    
   2017-01-19 19:00:32.689 :    CSSD:3469420288: clssnmFindVF: Duplicate voting disk found in the queue of previously configured disks 
   queued(o/192.168.10.18/PCW_CD_02_slcc05cel10|[66223efc-29254fbb-bf901601-21009 
   cbd]), 
   found(o/192.168.10.18/PCW_CD_02_slcc05cel10|[66223efc-29254fbb-bf901601-21009c 
   bd]), is not corrupted 
   2017-01-19 19:01:06.467 :    CSSD:3452057344: clssnmvVoteDiskValidation: 
   Voting disk(o/192.168.10.19/PCW_CD_02_slcc05cel11) is corrupted

3. You can also use SQL*Plus to confirm that the voting disks are corrupted:

   1. Log in as the grid user, and set the environment to ASM.

      [root@rmstest-udaau1 ~]# su - grid
      [grid@rmstest-udaau1 ~]$ . oraenv
      ORACLE_SID = [+ASM1] ? +ASM1
      The Oracle base has been set to /u01/app/grid

   2. Log in to SQL*Plus as SYSASM.

      $ORACLE_HOME/bin/sqlplus / as sysasm

   3. Run the following two queries:

      SQL> select name, voting_file from v$asm_disk where VOTING_FILE='Y' and group_number !=0;
      SQL> select  CC.name, count(*) from x$kfdat AA JOIN (select disk_number, name from v$asm_disk where VOTING_FILE='Y' and group_number !=0) CC ON CC.disk_number = AA.NUMBER_KFDAT where AA.FNUM_KFDAT= 1048572 group by CC.name;

      If the system is healthy, the results should look like the following example.

      Query 1 Results

      NAME                           VOTING_FILE
      ------------------------------ ---------------
      DBFSC3_CD_02_SLCLCX0788        Y
      DBFSC3_CD_09_SLCLCX0787        Y
      DBFSC3_CD_04_SLCLCX0786        Y

      Query 2 Results

      NAME                           COUNT(*)
      ------------------------------ ---------------
      DBFSC3_CD_02_SLCLCX0788        8
      DBFSC3_CD_09_SLCLCX0787        8
      DBFSC3_CD_04_SLCLCX0786        8

      In a healthy system, every voting disk returned in the first query should also be returned in the second query and the counts for all the disks should be non-zero. Otherwise, one or more of your voting disks are corrupted.

4. If a voting disks is corrupted, offline the grid disk that contains the voting disk. The cells will automatically move the bad voting disk to the other grid disk and online that voting disk.

   1. The following command offlines a grid disk named DATAC01_CD_05_SCAQAE08CELADM13.

      SQL> alter diskgroup DATAC01 offline disk DATAC01_CD_05_SCAQAE08CELADM13;
           Diskgroup altered.

   2. Wait 30 seconds and then rerun the two queries in step 3c to verify that the voting disk migrated to the new grid disk and that it is healthy.

   3. Verify the grid disk you offlined is now online:

      SQL> select name, mode_status, voting_file from v$asm_disk where name='DATAC01_CD_05_SCAQAE08CELADM13';

      The mode_status should be ONLINE, and the voting_file should NOT be Y.

   Repeat steps 4a through 4c for each remaining grid disk that contains a corrupt voting disk.

   Note
   
   

   If the CRS does not start because of the voting disk corruption, start it using Exclusive mode before you execute the command in step 4.

   crsctl start crs -excl

    

5. If you are using Oracle GI version 12.2.0.1 without any bundle patch, you must upgrade the GI version to the latest GI bundle, whether or not a voting disk was corrupted.

   See Patching Oracle Grid Infrastructure and Oracle Databases Using dbaascli for instructions on how to use the dbaascli utility to perform patching operations for Oracle Grid Infrastructure and Oracle Database on Exadata Database Service on Dedicated Infrastructure.

Direct link to this issue: Grid Infrastructure does not start after offlining and onlining a disk

Details: Exadata DB systems launched on June 15, 2018 or later automatically include the ability to create, list, and delete databases by using the Console, API, or Oracle Cloud Infrastructure CLI. However, systems provisioned before this date require extra steps to enable this functionality.

Attempts to use this functionality without the extra steps result in the following error messages:

• On creating a database - "Create Database is not supported on this Exadata DB system. To enable this feature, please contact Oracle Support."
• On terminating a database - "DeleteDbHome is not supported on this Exadata DB system. To enable this feature, please contact Oracle Support."

Workaround: You need to install the Exadata agent on each node of the Exadata DB system.

First, create a service request for assistance from Oracle Support Services. Oracle Support will respond by providing you with a preauthenticated URL for an Oracle Cloud Infrastructure Object Storage location where you can obtain the agent.

Before you install the Exadata agent:

• Upgrade the tooling (dbaastools rpm) to latest version on all the nodes on the Exadata DB system. See Updating Tooling on an Exadata Cloud Service Instance.
• Ensure that the system is configured to access Oracle Cloud Infrastructure Object Storage with the required security lists for the region in which the DB system was created. For more information about connectivity to Oracle Cloud Infrastructure Object Storage, see Prerequisites for Backups on Exadata Cloud Service.

To install the Exadata agent:

1. Log on to the node as root.

2. Run the following commands to install the agent:

   [root@<node_n>~]# cd /tmp
   [root@<node_n>~]# wget https://objectstorage.<region_name>.oraclecloud.com/p/1q523eOkAOYBJVP9RYji3V5APlMFHIv1_6bAMmxsS4E/n/dbaaspatchstore/b/dbaasexadatacustomersea1/o/backfill_agent_package_iwwva.tar
   [root@<node_n>~]# tar -xvf /tmp/backfill_agent_package_*.tar -C /tmp
   [root@<node_n>~]# rpm -ivh /tmp/dbcs-agent-2.5-3.x86_64.rpm

   Example output:

   [root@<node_n>~]# rpm -ivh dbcs-agent-2.5-3.x86_64.rpm
   Preparing...                ########################################### [100%]
   Checking for dbaastools_exa rpm on the system
   Current dbaastools_exa version = dbaastools_exa-1.0-1+18.1.4.1.0_180725.0000.x86_64
   dbaastools_exa version dbaastools_exa-1.0-1+18.1.4.1.0_180725.0000.x86_64 is good. Continuing with dbcs-agent installation
      1:dbcs-agent             ########################################### [100%]
   initctl: Unknown instance:
   initctl: Unknown instance:
   initzookeeper start/running, process 85821
   initdbcsagent stop/waiting
   initdbcsadmin stop/waiting
   initdbcsagent start/running, process 85833
   initdbcsadmin start/running, process 85836
   

3. Confirm that the agent is installed and running.

   [root@<node_n>~]# rpm -qa | grep dbcs-agent
   dbcs-agent-2.5-0.x86_64
   [root@<node_n>~]# initctl status initdbcsagent
   initdbcsagent start/running, process 97832

4. Repeat steps 1 through 3 on the remaining nodes.

After the agent is installed on all nodes, allow up to 30 minutes for Oracle to complete additional workflow tasks such as upgrading the agent to the latest version, rotating the agent credentials, and so on. When the process is complete, you should be able to use the Exadata managed features in the Console, API, or Oracle Cloud Infrastructure CLI.

Direct link to this issue: Managed features not enabled for systems provisioned before June 15, 2018

Details: The patching configuration file (/var/opt/oracle/exapatch/exadbcpatch.cfg) points to the object store of the us-phoenix-1 region, even if the Exadata DB system is deployed in another region.

This problem occurs if the release version of the database tooling package (dbaastools_exa) is 17430 or lower.

Workaround: Follow the instructions in Updating Tooling on an Exadata Cloud Service Instance to confirm that the release version of the tooling package is 17430 or lower, and then update it to the latest version.

Direct link to this issue: Patching configuration file points to wrong region

Details: A change in how Oracle Linux 7 handles temporary files can result in the removal of required socket files from the /var/tmp/.oracle directory. This issue affects only Exadata DB systems running the version 19.1.2 operating system image.

Workaround: Run sudo /usr/local/bin/imageinfo as the opc user to determine your operating system image version. If your image version is 19.1.2.0.0.190306, follow the instructions in Doc ID 2498572.1 to fix the issue.

Direct link to this issue: Various database workflow failures due to Oracle Linux 7 removal of required temporary files

Details: If you are using any of the OCI Java SDK synchronous clients that upload streams of data (for example ObjectStorageClient or DataSafeClient) and you do not define the RetryConfiguration at the client level or request level, you may be affected by silent data corruption.

Workaround: We are actively working on fixing this issue. For more information and possible workarounds, see the issue on GitHub.

Direct link to this issue: Potential data corruption issue in OCI Java SDK with binary data upload using default retries

Details: If you're using versions 2.14.1 and later of the OCI Java SDK, you may encounter performance regressions when using the SDK to call API operations on any of the OCI services. The regression causes a 30% to 60% increase in latency in SDK operations on any of the OCI services.

Workaround: We are actively working on fixing this issue. For more information and possible workarounds, see the issue on GitHub.

Direct link to this issue: Performance regression in OCI Java SDK versions 2.14.1 and later for all API operations

Details: Starting with version 2.0.0, the OCI SDK for Java supports using the Jersey ApacheConnectorProvider instead of the Jersey default HttpUrlConnectorProvider to allow the Apache HttpClient to make OCI service calls.

The ApacheConnectorProvider supports the use of Expect header by default for some Object Storage service operations (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/100). This has been observed to cause performance regression in the same Object Storage service operations.

final ApacheConnectorProperties apacheConnectorProperties =
        ApacheConnectorProperties.builder()
                .expectContinue(false) // disable the Expect header
                .build();

final ApacheConfigurator configurator =
        new ApacheConfigurator.NonBuffering(apacheConnectorProperties);
        
ObjectStorageClient objectStorageClient =
        ObjectStorageClient.builder()
                .clientConfigurator(configurator)
                .build(provider);

Direct link to this issue: Performance regression with the Apache Connector Provider in OCI SDK for Java

Details: In versions 2.0.0 to 2.13.0 of the OCI Java SDK API, some operations that return a stream of data but don't return the content-length header in the response might return truncated data. This is caused by the SDK prematurely closing the stream before reading all the data.

Workaround:Update the OCI Java SDK client to version 2.13.1 or later. For more information about this issue and workarounds, see https://github.com/oracle/oci-java-sdk/issues/357

Direct link to this issue: Truncated response for operations that return binary data with the OCI Java SDK

Details: Due to some issues with one of its dependencies, the Go SDK feature which allows customers to automatically use new realms which might be unknown to the SDK is not functioning from within Cloud Shell.

can not create client, bad configuration: failed to get security token: failed to renew security token: failed to get security token: failed to call: Post "https://<endpoint>/v1/x509": dial tcp: lookup <endpoint> on 127.0.0.11:53: server misbehaving
panicked while retrying operation. Panic was: runtime error: invalid memory address or nil pointer dereference

Workaround: To resolve this issue, enable resolving regions using the instance metadata service for Go SDK. For more information, see: Adding Regions

Direct link to this issue: Go SDK cannot automatically find some regions while running in Cloud Shell

Details: You may encounter an increase in latency for operations made to some OCI services using the SDKs, Terraform, Ansible, and the CLI. This issue has been confirmed to impact the OCI Streaming service, and likely impacts the Email Delivery, Health Checks, NoSQL Database Cloud, Registry, Generic Artifacts, and Web Application Acceleration and Security services as well. This list is not comprehensive, and it is possible you may encounter the issue against other OCI services as well. The issue has been confirmed to NOT affect the OCI Object Storage and Functions services.

The following SDKs and tools are impacted:

Workarounds and more information: We are actively working on fixing this issue. For more information and possible workarounds, please see the following:

Direct link to this issue: Increased latency issues in operations for some OCI services using the SDKS and other tools

Details: The copy_boot_volume_backup_and_wait_for_state and copy_volume_backup_and_wait_for_state from the BlockStorage Client Composite operations throw a 404/NotAuthorizedOrNotFound when copying a backup from one region to another. For more information see: https://github.com/oracle/oci-python-sdk/issues/344.

Workaround: Instead of using the composite operations, use two different clients for this operation; one client in the Source Region to send the request for copying the backup from Region A to Region B, and a second client in Destination region to wait for the backup to become available. See example here: https://github.com/oracle/oci-python-sdk/blob/master/examples/copy_volume_backup_example.py

Direct link to this issue: Python SDK composite operations throw a 404 NotAuthorizedOrNotFound error even though operation is a success

Details: The OCI SDK for TypeScript and JavaScript have a known issue with big numbers greater than JavaScript's Number.MAX_SAFE_INTEGER. Any Numbers greater than Number.MAX_SAFE_INTEGER will result in rounding issue.

Workaround: We are aware of the issue where an API response may send back a number greater than JavaScript's Number.MAX_SAFE_INTERGER. At the moment the number rounding issue is not an impact to calling any APIs.

Direct link to this issue: Potential data rounding issue for big numbers with OCI SDK for TypeScript and JavaScript

Details: When using version 1.25.1 or earlier of the OCI Java SDK clients that upload streams of data (for example ObjectStorageClient or FunctionsInvokeClient), either synchronously and asynchronously, and you use a RefreshableOnNotAuthenticatedProvider (for example, for Resource Principals or Instance Principals) you may be affected by silent data corruption.

Workaround: Update the OCI Java SDK client to version 1.25.2 or later. For more information about this issue and workarounds, see Potential data corruption issue for OCI Java SDK on binary data upload with RefreshableOnNotAuthenticatedProvider.

Direct link to this issue: Potential data corruption issue with OCI Java SDK on binary data upload with RefreshableOnNotAuthenticatedProvider

Details: If you are using version 3.2.1.1 or earlier of the OCI HDFS Connector clients and you use a RefreshableOnNotAuthenticatedProvider (e.g. InstancePrincipalsCustomAuthenticator, or generally for Resource Principals or Instance Principals) you may be affected by silent data corruption.

Workaround: Update the OCI HDFS Connector client to version 3.2.1.3 or later. For more information about this issue and workarounds, see Potential data corruption issue for OCI HDFS Connector with RefreshableOnNotAuthenticatedProvider.

Direct link to this issue: Potential data corruption issue with OCI HDFS Connector on binary data upload with RefreshableOnNotAuthenticatedProvider

Details: When using the SDK for Python to perform binary upload operations you may encounter an issue with data corruption if retries are enabled or if you are using UploadManager.upload_file.

Workaround: We're working on a resolution. For more information about this issue and workarounds, see Potential data corruption issue for PythonSDK retry on binary data upload.

Direct link to this issue: Potential data corruption with SDK for Python on binary upload

Update: The root cause of the issue causing data corruption has been fixed with the release of v2.54.0. Please use oci v2.54.0 or above to avoid data corruption. The behavior of older versions of the OCI Python SDK regarding this issue has been explained below.

Details: Customers using the OCI SDK for Python and oci.object_storage.UploadManager.upload_stream in FIPS mode might be vulnerable to silent data corruption. If the circumstances to produce the issue are true for your environment, the SDK reports success for the upload operation, but a 0-length object is uploaded.

The resolution differs depending on the state of your environment:

1. Using UploadManager.upload_stream() in an environment which uses a FIPS-compliant OpenSSL version where the SDK for Python is not operating in FIPS mode as described in Using FIPS validated libraries.

   To determine if you fall under this scenario:

   • Verify that you are using a FIPS-compliant OpenSSL version by running the command openssl version . If "fips" is part of the version name, and you are not operating the SDK in FIPS mode, then you would fall under this scenario.

   • If oci.fips.is_fips_mode() does not return True, then the SDK is not operating in FIPS mode.

   Workaround: Upgrade the OCI SDK for Python to version 2.53.1 or greater and operate the SDK for Python in FIPS mode as described in Using FIPS validated libraries.

   Important
   
   Not operating the SDK in FIPS mode while using a FIPS-compliant OpenSSL version will still result in data corruption while using UploadManager.upload_stream().
2. Using UploadManager.upload_stream() when the SDK for Python is operating in FIPS mode as described in Using FIPS validated libraries and the SDK for Python is v2.53.0 or lower.

   If oci.fips.is_fips_mode() returns True, then the SDK is operating in FIPS mode.

   Resolution: Upgrade the OCI SDK for Python to version 2.53.1 or greater.

For more information about this issue, see Potential data corruption issue for multipart stream upload for OCI Python SDK on GitHub.

Direct link to this issue: Potential data corruption issue with SDK for Python and upload streams

Details: If you use volume group backups when performing DR operations for compute and storage across different ADs within the same region, back and forth DR transitions will cause the compute and associated block storage (which uses volume group backups) to end up in a different AD each time.

Workaround: This issue does not affect block storage that is replicated using volume group replication.

Details: Auto-tune performance settings for block storage volumes are not carried over during DR operations.

Workaround: For block storage volumes which have auto-tuned performance enabled you must re-enable these settings after FSDR transitions these block storage volumes to another region.

Details: If you perform a failover operation immediately after modifying an FSDR-protected resource, then the resource recovery may fail, or the resource may not be recovered properly. For example, if you change the replication target or other properties for a volume group that you added to a DR protection group, and the primary region suffers an immediate outage thereafter, FSDR may not detect the changes you made to the volume group replication settings, and this will affect recovery of that volume group.

Workaround: Perform a switchover precheck immediately after making any changes to any resources under DR protection.

Details: FSDR uses the Oracle Cloud Agent (OCA) Run Command utility to run local scripts on instances. When you configure a user-defined step to run a local script on a Microsoft Windows instance, then you cannot use the FSDR Run As User feature that allows you to specify a different userid to run local scripts that reside on instances.

Workaround: On Microsoft Windows instances, the script can only run as the default ocarun userid used by the Oracle Cloud Agent Run Command utility. This limitation does not affect Oracle Linux instances.

Details: FSDR uses the Oracle Cloud Agent (OCA) Run Command utility to run local scripts on instances. By default, these scripts are run as the ocarun user.

Workaround: On a Microsoft Windows instance, any local script that you configure to run as a user-defined step in a DR plan must be accessible and executable by this ocarun userid.

Details: When running a local script using a user-defined step in a DR plan, if you do not provide full paths to script interpreters or scripts, then FSDR will throw errors.

Workaround: When you configure a user-defined step in a DR plan to run a local script that resides on an instance, ensure that you provide the full path to any interpreter that may precede the script name, as well as the full path to the script.

Details: During DR operations, FSDR attempts to reassign the original private IP assigned to an instance if the CIDR-block of the destination subnet matches the CIDR-block of the source subnet, and if the original private IP of the instance is not already assigned.

If you use FSDR to relocate all the nodes in an OCFS2 cluster, and the private IP for any of the cluster node cannot be reassigned, those cluster nodes will detach from the OCFS2 cluster after the nodes are launched in the standby region.

Workaround: Ensure that the destination subnet's CIDR-block matches the CIDR-block of the source subnet and all private IP addresses required for cluster nodes are available in the destination subnet.

Details: After FSDR relocates an instance to a different region, the resource page of the instance may display the following message for Instance Access:

We are not quite sure how to connect to an instance this uses this image

Workaround: Ignore this message. SSH connections to the instance will work normally if you use the original SSH keyfile to connect to and authenticate the instance.

Details: After FSDR relocates an instance to a different region, the resource page of the instance may display incorrect information for the Image portion of its boot volume.

For example, the Image information column may display the following message: Error loading data

Workaround: This error message is for the display of the Image name but does not affect the operation of the instance or its boot volume.

Details: The on-demand upload of a zip file which is created on a Windows machine might sometimes fail to upload the log content. The reason for the failure is that the zip created on Windows has the same last modification time as the file's creation time. So, when the file is unzipped, the file's last modification time is set as the file's creation time which might be older than the timestamp of the log entries in the log file. In such a case, the log entries with the timestamp more recent than the file's last modification time are not uploaded.

An example of the issue:

Timestamp on the log entry: 2020-10-12 21:12:06

File last modification time of the log file: 2020-10-10 08:00:00

Workaround: Copy the log files to a new folder and create a zip file. This action makes the file's last modification time more recent than the timestamp of the log entries. Use this new zip file for on-demand upload.

Using the previous example, after the workaround is implemented:

Timestamp on the log entry: 2020-10-12 21:12:06

File last modification time of the log file: 2021-03-31 08:00:00

Direct link to this issue: On-demand upload from a Windows machine using a zip file

Details: Folders containing more than 10,000 files can cause log collection issues (as well as operating system issues).

When large folders are encountered by the Management Agent Logging Analytics plug-in, a message similar to the following example message is added to the Management Agent mgmt_agent.log file:

2020-07-30 14:46:51,653 [LOG.Executor.2388 (LA_TASK_os_file)-61850] INFO - ignore large dir /u01/service/database/logs. set property loganalytics.enable_large_dir to enable.

Resolution: We recommend avoiding large folders.

However, if you want to continue monitoring logs in large folders, then you can enable the property indicated in the mgmt_agent.log file by performing the following action:

sudo -u mgmt_agent echo "loganalytics.enable_large_dir=true" >> INSTALL_DIRECTORY/agent_inst/config/emd.properties

Replace INSTALL_DIRECTORY with the path to the agent_inst folder.

Direct link to this issue: Special handling when monitoring logs in large folders

Details: Up to now, you might have been using either the tenancy name or the tenancy namespace when logging in to Oracle Cloud Infrastructure Registry and when performing operations on images in the Container Registry.

After September 30, 2019, you will have to use the tenancy namespace rather than the tenancy name when using Oracle Cloud Infrastructure Registry.

Background: After September 30, 2019, you will not be able to:

• Specify the tenancy name when logging in to Oracle Cloud Infrastructure Registry.
• Perform operations on images that include tenancy name in the repository path.

Instead, you will have to use the tenancy namespace rather than the tenancy name when using Oracle Cloud Infrastructure Registry.

A tenancy namespace is an auto-generated and immutable random string of alphanumeric characters. For example, the namespace of the acme-dev tenancy might be ansh81vru1zp. You can see the tenancy namespace on the Container Registry page of the Console.

Note that for some older tenancies, the tenancy namespace might be the same as the tenancy name. If that is the case, no action is required.

On or before September 30, 2019, if the tenancy namespace and the tenancy name are different, you must:

• Start specifying the tenancy namespace when logging in to Oracle Cloud Infrastructure Registry, instead of the tenancy name.
• Start specifying the tenancy namespace when pushing new images to Oracle Cloud Infrastructure Registry, instead of the tenancy name.
• Migrate any existing images in Oracle Cloud Infrastructure Registry that include the tenancy name in the path.

The following workarounds and examples assume:

• tenancy name is acme-dev
• tenancy namespace is ansh81vru1zp
• username is jdoe@acme.com

Workaround for logging into Oracle Cloud Infrastructure Registry: Previously, when you logged in to Oracle Cloud Infrastructure Registry and were prompted for a username, you could have entered it in the format <tenancy-name>/<username>.

For example:

$ docker login phx.ocir.io

Username: acme-dev/jdoe@acme.com
Password:

On or before September 30, 2019, you must start using the tenancy namespace instead of the tenancy name when logging in to Oracle Cloud Infrastructure Registry. When you are prompted for username, enter it in the format <tenancy-namespace>/<username>.

For example:

$ docker login phx.ocir.io

Username: ansh81vru1zp/jdoe@acme.com
Password:

Workaround for pushing new images to Oracle Cloud Infrastructure Registry: Previously, when you pushed a new image to Oracle Cloud Infrastructure Registry, you could have specified the tenancy name as part of the repository path in the docker push command. You could have entered the command in the format:

$ docker push <region-key>.ocir.io/<tenancy-name>/<image-name>:<tag>

For example:

$ docker push phx.ocir.io/acme-dev/helloworld:latest

On or before September 30, 2019, you must start using the tenancy namespace instead of the tenancy name in the docker push command when you push new images. Enter the command in the format:

$ docker push <region-key>.ocir.io/<tenancy-namespace>/<image-name>:<tag>

For example:

$ docker push phx.ocir.io/ansh81vru1zp/helloworld:latest

Workaround for existing images in Oracle Cloud Infrastructure Registry that include the tenancy name in the repository path: If you have previously pushed images to Oracle Cloud Infrastructure Registry, those existing images could have included the tenancy name as part of the repository path. For example, phx.ocir.io/acme-dev/helloworld:latest.

After September 30, 2019, you will not be able to perform operations on existing images in the Container Registry that include the tenancy name in the repository path.

So on or before September 30, 2019, for every existing image that contains the tenancy name in the repository path, you must replace tenancy name with tenancy namespace.

To replace tenancy name with tenancy namespace in the repository path of an existing image:

1. Pull the image by entering:

   $ docker pull <region-key>.ocir.io/<tenancy-name>/<image-name>:<tag>

   For example:

   $ docker pull phx.ocir.io/acme-dev/helloworld:latest

2. Use the docker tag command to change the repository path by entering:

   $ docker tag <region-key>.ocir.io/<tenancy-name>/<image-name>:<tag> <region-key>.ocir.io/<tenancy-namespace>/<image-name>:<tag>

   For example:

   $ docker tag phx.ocir.io/acme-dev/helloworld:latest phx.ocir.io/ansh81vru1zp/helloworld:latest

3. Push the image with the new repository path to the Container Registry by entering:

   $ docker push <region-key>.ocir.io/<tenancy-namespace>/<image-name>:<tag>

   For example:

   $ docker push phx.ocir.io/ansh81vru1zp/helloworld:latest

4. Repeat the above steps for every existing image that has tenancy name in the repository path.

Direct link to this issue: Use Tenancy Namespace instead of Tenancy Name in image tags and Docker login credentials on or before September 30, 2019