Rotating Cluster Credentials

Find out how to rotate the credentials of clusters you've created using Container Engine for Kubernetes (OKE).

To enable secure TLS (formerly SSL) communication to, from, and within clusters, Kubernetes uses PKI (Public Key Infrastructure) certificates for authentication. For more information about PKI certificates and Kubernetes clusters, see PKI certificates and requirements in the Kubernetes documentation.

The clusters you create with Container Engine for Kubernetes have PKI root CAs (Certificate Authorities) and private keys for the cluster, the etcd component, and the front-proxy component. These root CAs are used to sign the certificates and private keys used in the cluster. The root CAs, signed certificates, and private keys used in the cluster are collectively known as cluster credentials.

In clusters you create with Container Engine for Kubernetes, the root CAs expire after five years. To continue using a cluster, you have to change (or 'rotate') cluster credentials before the end of the five year period. Additionally, your organization might have security guidelines and policies in place that require you to rotate cluster credentials more frequently (in some cases, much more frequently). Messages informing you of upcoming cluster credential expiry are shown in the Console. You can also use the Console, the CLI, and the API to find out when cluster credentials are due to expire.

When you rotate cluster credentials, you have to update Kubernetes API clients that were using the previous credentials to communicate with the Kubernetes API. Such Kubernetes API clients include kubeconfig files, and pods that communicate directly with the Kubernetes API.

Rotating cluster credentials is a three-step process:

• Step 1, Initiate the Start Credential Rotation phase: During the Start Credential Rotation phase, new root CAs, private keys, and certificates are created. You can initiate the Start Credential Rotation phase at any time, but you must start credential rotation before the credentials expire. Starting credential rotation before the credentials expire will also avoid service interruption.

  When initiating the Start Credential Rotation phase, you explicitly specify the length of a 'delay period' before the Complete Credential Rotation phase is initiated automatically. The delay period enables you to update Kubernetes API clients (Step 2). During the delay period, both the legacy credentials and the new credentials provide secure communication to, from, and within the cluster. You can specify any length of time for the delay period, up to 14 days. You can manually initiate the Complete Credential Rotation phase before the end of the delay period, provided the new credentials are at least 24 hours old and you have successfully completed Step 2.

  Do not start Step 2 until the new root CAs, private keys, and certificates have been created.

  Note
  
  When using the CLI or the API to initiate the Start Credential Rotation phase, specify the length of the delay period in the ISO 8601 duration format. For example:

  • use P5D to specify five days
  • use PT5H to specify five hours
  • use PT5M to specify five minutes

  For more information about the ISO 8601 duration format, search online.

• Step 2, Update Kubernetes API clients: When the new root CAs, private keys, and certificates have been created, and within the delay period you specified (any length of time, up to 14 days):
  • Set up new kubeconfig files to enable access to the cluster using both the new credentials and the legacy credentials.
  • Recreate or restart worker nodes hosting pods that communicate directly with the Kubernetes API Server (or to avoid interruption to the worker nodes, simply restart the pods themselves).

  Do not start Step 3 until you have set up new kubeconfig files, and have recreated or restarted worker nodes (or simply restarted the pods).

• Step 3, Initiate the Complete Credential Rotation phase: When you have finished Step 2, and within the delay period you specified (any length of time, up to 14 days), you can manually initiate the Complete Credential Rotation phase. During the Complete Credential Rotation phase, the legacy root CAs, private keys, and certificates are retired and removed. Initiate the Complete Credential Rotation phase no sooner than 24 hours after you initiated the Start Credential Rotation phase, and only after you have finished step 2. Provided the new credentials are at least 24 hours old, you can initiate the Complete Credential Rotation phase as soon as you are ready.

  If you do not manually initiate the Complete Credential Rotation phase during the delay period you specified (any length of time, up to 14 days), the Complete Credential Rotation phase is automatically initiated at the end of the delay period.

  When the Complete Credential Rotation phase is complete, we recommend replacing any kubeconfig file created during step 2 (which will contain both legacy credentials and new credentials) with a new kubeconfig file containing only the new credentials.

The Start Credential Rotation phase and Complete Credential Rotation phase are spawned as separate work requests. To track progress of the Start Credential Rotation phase and Complete Credential Rotation phase, view the status of the corresponding work request. See Viewing Work Requests.

You can rotate cluster credentials using the Console, the CLI, and the API.

Note the following when rotating cluster credentials:

• Simply upgrading the Kubernetes version on the cluster control plane does not rotate credentials. You have to complete all the steps in the credential rotation process.
• The Start Credential Rotation phase and Complete Credential Rotation phase can only begin when all worker nodes in the cluster have a status of READY. If the status of one or more worker nodes changes to NOT READY during credential rotation, the credential rotation operations pause for up to 24 hours, waiting for all nodes to again have a status of READY before resuming. If one or more worker nodes still has a status of NOT READY after 24 hours, the credential rotation operations time out.
• If the Start Credential Rotation phase or the Complete Credential Rotation phase fails for some reason, the legacy credentials continue to work and are not retired until they expire.
• If you do not manually initiate the Complete Credential Rotation phase within the delay period you specified (any length of time, up to 14 days), the Complete Credential Rotation phase is initiated automatically. If you have not updated Kubernetes API clients before the end of the delay period, the cluster will experience service interruption.
• Credential rotation is supported for clusters with managed nodes, and for clusters with self-managed nodes. Credential rotation is not yet supported for clusters with virtual nodes.