Tagging Kubernetes Cluster-Related Resources

You can use Oracle Cloud Infrastructure tags to organize, control, manage, and report on cloud resources used by the Kubernetes clusters you create with Container Engine for Kubernetes. Tagging enables you to group disparate resources across compartments, and also enables you to annotate resources with your own metadata.

Typically, you'll find tagging useful for:

  • Resource tracking: For example, to:

    • report all Oracle Cloud Infrastructure resources (compute instances, load balancers, block volumes) associated with a Kubernetes cluster
    • apply application-specific tags to load balancers and block volumes to track resources for a given application running in a cluster
  • Cost tracking: For example, to obtain detailed cost reports for the running of Kubernetes clusters by application, or by line of business.

  • Authorization grouping: For example, to authorize managed nodes to access a designated object storage bucket (using instance principals for worker nodes as part of a node pool dynamic group tag).

Summary of Tagging Concepts for Kubernetes Clusters

The Oracle Cloud Infrastructure Tagging service documentation describes how tagging works in detail. This topic summarizes some of the key concepts you need to be familiar with when tagging cluster-related resources you create with Container Engine for Kubernetes.

Defined tags

Defined tags are set up and managed by a tag administrator. A defined tag consists of a tag namespace, a key, and a value. The tag namespace and tag key definition must be set up in a tenancy before you can apply a defined tag to a resource.

The tag administrator can set up a defined tag key with either a tag value type of string (enabling you to add a value, or leave blank), or with a list of predefined values (from which you must choose a value). The tag administrator can specify tag variables to set the value of a defined tag.

You can apply a defined tag when creating a new resource, or when updating an existing resource. When applying a defined tag, you select the defined tag from a list of defined tags.

For more information, see Tags and Tag Namespace Concepts.

Free-form tags

Free-form tags are not managed by a tag administrator. A free-form tag consists of a key and a value. Unlike defined tags, free-form tags do not belong to a tag namespace.

Free-form tags cannot be set up with pre-defined values, or with tag variables. You cannot include free-form tags in IAM policies.

You can apply a free-form tag when creating a new resource, or when updating an existing resource. When applying a free-form tag, you don't see a list of existing free-form tags from which to select.

For more information, see Understanding Free-form Tags.

Tag Defaults

Tag defaults are set up and managed for a specific compartment. Tag defaults are compartment-specific. Tag defaults specify tags that are applied automatically to all resources created in a specific compartment at the time of creation. A maximum of five tag defaults can be set up for each compartment.

Tag defaults must have a tag value. When a tag default is set up, the value of the tag default is specified as one of the following:
  • a default value, which is always applied to all resources created in the compartment, with no input from the user creating the resource
  • a user-applied value, which the user creating a resource in the compartment must enter at the time of resource creation

Note that if tag defaults are updated or deleted, existing tags that have already been applied to resources are not modified.

For more information, see Managing Tag Defaults.

Cost-Tracking Tags

All defined tags can be used for Cost Analysis and in Cost and Usage Reports. However, you can also designate one or more defined tags as cost-tracking tags, for use when setting budgets. Cost-tracking tags enable you to set soft limits on your Oracle Cloud Infrastructure spending. You can set alerts on your budget to let you know when you might exceed your budget, and you can view all of your budgets and spending from a single place in the Console.

For more information, see Using Cost-Tracking Tags.

Applying Tags to Cluster-Related Resources

You can apply tags to all cluster-related resources.

If you use the 'Custom Create' workflow to create a new cluster, you can specify defined and free-form tags to apply to cluster-related resources (the cluster itself, node pools, nodes, load balancers, and block storage volumes). In the case of load balancers and block storage volumes, you can override these 'initial' tags using annotations and parameters in manifest files.

If you use the 'Quick Create' workflow to create a new cluster, a free-form tag is automatically added to all the resources that are created (to the cluster itself, as well as to any new VCNs, subnets, security lists, internet gateways, service gateways, and NAT gateways). In addition, a free-form tag is also automatically added to new managed node pools and managed nodes (compute instances) created as part of the 'Quick Create' workflow. These free-form tags are not added to resources if you use the 'Custom Create' workflow to create the cluster.

When tags are applied to cluster-related resources, you can use the tags to filter the resources you see in the Console. For more information, see Filtering the Displayed List of Resources.

For more information about applying tags to particular types of resources, see:

Applying Tags to Clusters

When you create a cluster, you can optionally apply tags to the cluster resource. These tags are referred to as cluster tags. You can specify both defined tags and free-form tags as cluster tags.

Tag defaults with default values that are specified for the compartment are automatically applied to the cluster resource as well. Note that Container Engine for Kubernetes does not currently support tag defaults with user-applied values.

Note the following:

  • If you use the 'Quick Create' workflow to create a new cluster, the free-form tag "OKEclusterName": <cluster-name> is automatically added to the cluster. The tag is not added to a cluster if you use the 'Custom Create' workflow to create the cluster. Note that the value of this tag does not change if you subsequently change the name of the cluster.
  • Tags applied to a cluster resource are not applied to either the cluster's private IP address resource or to the optional public IP address resource.
  • To apply defined tags from a tag namespace belonging to one compartment to a cluster resource belonging to a different compartment, you must include a policy statement to allow the cluster to use the tag namespace. See Additional IAM Policy when a Cluster and a Tag Namespace are in Different Compartments.

Using the Console to Specify Cluster Tags

To add a cluster tag to a new cluster:

  1. Follow the instructions in Using the Console to create a Cluster with Explicitly Defined Settings in the 'Custom Create' workflow to create a new cluster.
  2. Display the Cluster tags section of the Create Cluster page.
  3. To add a defined tag to the cluster:
    • Tag Namespace: Select the tag namespace to which the tag belongs.
    • Tag Key: Select the name of the defined tag to apply to the cluster.
    • Tag Value: Either select the value for the tag from a pre-defined list of values, or enter a new value, or leave blank (depending on how the defined tag has been set up).
  4. To add a free-form tag to the cluster:
    • Tag Namespace: Set to None (free-form tags do not belong to a tag namespace).
    • Tag Key: Enter a name for the free-form tag to apply to the cluster.
    • Tag Value: Enter a value for the tag to apply to the cluster.

To update cluster tags applied to an existing cluster:

  1. Follow the instructions in Modifying Kubernetes Cluster Properties to update an existing cluster.
  2. Display the Tags tab of the Cluster Details page.
  3. Click Add Tags to add, remove, and change the value of defined tags and free-form tags applied to the cluster.

Using the CLI to Specify Cluster Tags

oci ce cluster create \
--compartment-id <compartment-ocid> \
--kubernetes-version <kubernetes-version> \
--name <cluster-name> --vcn-id <vcn-ocid> \
--defined-tags <json-name-value-pairs> \
--freeform-tags <json-name-value-pairs>

For example:

oci ce cluster create \
--compartment-id ocid1.compartment.oc1..aaaaaaaay______t6q \
--kubernetes-version v1.20.11 \
--name Finance-Cluster \
--vcn-id ocid1.vcn.oc1.iad.aaaaaae___yja \
--defined-tags {"Operations": {"CostCenter": "42"}} \
--freeform-tags {"Department": "Finance"}

Using the API to Specify Cluster Tags

Use the freeformTags and definedTags attributes of the CreateClusterDetails and UpdateClusterDetails objects used by the CreateCluster and Update Cluster operations to add and update cluster tags.

Applying Tags to Node Pools

When you create a managed node pool or a virtual node pool, you can optionally apply tags to the node pool resource. These tags are referred to as node pool tags. You can specify both defined tags and free-form tags as node pool tags.

Tag defaults with default values that are specified for the node pool's compartment are automatically applied to the node pool resource. Tag defaults with user-applied values are only applied to the node pool resource if the node pool is in a different compartment to the cluster. If that is the case, then you must specify a value for tag defaults with user-applied values by specifying node pool tags, as described in this section.

Note the following:

  • If you use the 'Quick Create' workflow to create a new cluster, the free-form tags "OKEclusterName": <cluster-name> and "OKEnodePoolName": <node-pool-name> are also automatically added to node pools in the new cluster. The tags are not added to node pools if you use the 'Custom Create' workflow to create the cluster. Note that the values of these tags do not change if you subsequently change the name of the cluster or node pool.
  • Node pool tags are not applied to worker nodes in the node pool. To apply tags to worker nodes, see Applying Tags to Nodes.
  • To apply defined tags from a tag namespace belonging to one compartment to a node pool belonging to a different compartment, you must include a policy statement to allow the cluster to use the tag namespace. See Additional IAM Policy when a Cluster and a Tag Namespace are in Different Compartments.

Using the Console to Specify Node Pool Tags

To add a node pool tag to a new node pool when creating a new cluster:

  1. Follow the instructions in Using the Console to create a Cluster with Explicitly Defined Settings in the 'Custom Create' workflow to create a new cluster.
  2. Display the Node Pool tags section of the Node Pools page.
  3. To add a defined tag to the node pool:
    • Tag Namespace: Select the tag namespace to which the tag belongs.
    • Tag Key: Select the name of the defined tag to apply to the node pool.
    • Tag Value: Either select the value for the tag from a pre-defined list of values, or enter a new value, or leave blank (depending on how the defined tag has been set up).
  4. To add a free-form tag to the node pool:
    • Tag Namespace: Set to None (free-form tags do not belong to a tag namespace).
    • Tag Key: Enter a name for the free-form tag to apply to the node pool.
    • Tag Value: Enter a value for the tag to apply to the node pool.

To update node pool tags applied to an existing node pool:

  1. Follow the instructions in Modifying Node Pool and Worker Node Properties to update an existing node pool.
  2. Display the Tags tab of the Node Pool Details page.
  3. Click Add Tags to add, remove, and change the value of defined tags and free-form tags applied to the node pool.

Using the CLI to Specify Managed Node Pool Tags

oci ce node-pool create \
--cluster-id <cluster-ocid> \
--compartment-id <compartment-ocid> \
--kubernetes-version <kubernetes-version> \
--name <node-pool-name> \
--node-shape <node-shape> \
--defined-tags <json-name-value-pairs> \
--freeform-tags <json-name-value-pairs>

For example:

oci ce node-pool create \
--cluster-id ocid1.cluster.oc1.iad.aaaaaaaaaf______jrd \
--compartment-id ocid1.compartment.oc1..aaaaaaaay______t6q \
--kubernetes-version v1.20.11 \
--name Finance-Node-Pool \
--node-shape VM.Standard2.1 \
--defined-tags {"Operations": {"CostCenter": "42"}} \
--freeform-tags {"Department": "Finance"}

Using the CLI to Specify Virtual Node Pool Tags

oci ce virtual-node-pool create \
--cluster-id <cluster-ocid> \
--compartment-id <compartment-ocid> \
--display-name <node-pool-name> \
--kubernetes-version <kubernetes-version> \
--placement-configurations "[{\"availabilityDomain\":\"<ad-name>\",\"faultDomain\":[\"FAULT-DOMAIN-<n>\"],\"subnetId\":\"<virtualnode-subnet-ocid>\"}]" \
--nsg-ids "[\"<virtual-node-nsg-ocid>\"]" \
--pod-configuration "{\"subnetId\":\"<pod-subnet-ocid>\",\"nsgIds\":[\"<pod-nsg-ocid>\"],\"shape\":\"<shape-name>\"}" \
--size <number-of-nodes> \
--defined-tags <json-name-value-pairs> \
--freeform-tags <json-name-value-pairs>

For example:

oci ce virtual-node-pool create \
--cluster-id ocid1.cluster.oc1.phx.aaaaaaaa______w5q \
--compartment-id ocid1.compartment.oc1..aaaaaaaa______n5q \
--display-name sales-vnp \
--kubernetes-version v1.24.1 \
--placement-configurations "[{\"availabilityDomain\":\"GMvH:PHX-AD-1\",\"faultDomain\":[\"FAULT-DOMAIN-1\"],\"subnetId\":\"ocid1.subnet.oc1.phx.aaaaaaaa______sra\"}]" \
--nsg-ids "[\"ocid1.networksecuritygroup.oc1.phx.aaaaaaaa______hpa\"]" \
--pod-configuration "{\"subnetId\":\"ocid1.subnet.oc1.phx.aaaaaaaa______o7q\",\"nsgIds\":[\"ocid1.networksecuritygroup.oc1.phx.aaaaaaaa______osq\"],\"shape\":\"Pod.Standard.E4.Flex\"}" \
--size 1 \
--defined-tags {"Operations": {"CostCenter": "43"}} \
--freeform-tags {"Department": "Sales"}

Using the API to Specify Node Pool Tags

To add and update managed node pool tags, use the freeformTags and definedTags attributes of the CreateNodePoolDetails and UpdateNodePoolDetails objects used by the CreateNodePool and UpdateNodePool operations.

To add and update virtual node pool tags, use the freeformTags and definedTags attributes of the CreateVirtualNodePoolDetails and UpdateVirtualNodePoolDetails objects used by the CreateVirtualNodePool and UpdateVirtualNodePool operations.

Applying Tags to Nodes

When you define a node pool, you can optionally apply tags to the worker nodes in the node pool. These tags are referred to as node tags. You can specify both defined tags and free-form tags as node tags.

When you define a managed node pool, any node tags you apply are applied to the compute instances hosting the managed nodes.

Tag defaults with default values that are specified for the corresponding node pool's compartment are automatically applied to worker nodes. Tag defaults with user-applied values are only applied if the node pool is in a different compartment to the cluster. If that is the case, then you must specify a value for tag defaults with user-applied values by specifying node tags, as described in this section.

Note the following:

  • If you use the 'Quick Create' workflow to create a new cluster, the free-form tags "OKEclusterName": <cluster-name> and "OKEnodePoolName": <node-pool-name> are also automatically added to worker nodes in the new cluster. The tags are not added to worker nodes if you use the 'Custom Create' workflow to create the cluster. Note that the tag values do not change if you subsequently change the name of the cluster or node pool.
  • If you specify node tags when defining a node pool and the node pool is subsequently scaled out, the node tags are automatically applied to any new worker nodes that are created.
  • Tags are only applied to worker nodes when the worker nodes are first created. If you update the tags to be applied to worker nodes, the changes only apply to new worker nodes. Tags applied to existing worker nodes are unaffected.
  • To apply defined tags from a tag namespace belonging to one compartment to worker nodes in a node pool belonging to a different compartment, you must include a policy statement to allow the cluster to use the tag namespace. See Additional IAM Policy when a Cluster and a Tag Namespace are in Different Compartments.
  • If you apply a cost-tracking defined tag to worker nodes, you can include worker node compute instance usage in budgets (see Using Cost-Tracking Tags).

Using the Console to Specify Managed Node Tags

To add a node tag to managed nodes in a new managed node pool when creating a new cluster:

  1. Follow the instructions in Using the Console to create a Cluster with Explicitly Defined Settings in the 'Custom Create' workflow to create a new cluster.
  2. Display the Node tags section of the Node Pools page.
  3. To add a defined node tag to worker nodes in the new node pool:
    • Tag Namespace: Select the tag namespace to which the tag belongs.
    • Tag Key: Select the name of the defined tag to apply to worker nodes.
    • Tag Value: Either select the value for the tag from a pre-defined list of values, or enter a new value, or leave blank (depending on how the defined tag has been set up).
  4. To add a free-form node tag to worker nodes in the new node pool:
    • Tag Namespace: Set to None (free-form tags do not belong to a tag namespace).
    • Tag Key: Enter a name for the free-form tag to apply to worker nodes.
    • Tag Value: Enter a value for the tag to apply to worker nodes.

To update node tags applied to new managed nodes created in an existing managed node pool:

  1. Follow the instructions in Modifying Node Pool and Worker Node Properties to update an existing node pool.
  2. Display the Node Tags tab of the Node Pool Details page.
  3. Click Add Tags to add, remove, and change the value of defined tags and free-form tags applied to worker nodes in the node pool.

    Note that tags are only applied to worker nodes when the worker nodes are first created. So if you update the tags applied to worker nodes, the changes only apply to new worker nodes. Tags already applied to existing worker nodes are unaffected.

Using the CLI to Specify Managed Node Tags

oci ce node-pool create \
--cluster-id <cluster-ocid> \
--compartment-id <compartment-ocid> \
--kubernetes-version <kubernetes-version> \
--name <node-pool-name> \
--node-shape <node-shape> \
--node-defined-tags <json-name-value-pairs> \
--node-freeform-tags <json-name-value-pairs>

For example:

oci ce node-pool create \
--cluster-id ocid1.cluster.oc1.iad.aaaaaaaaaf______jrd \
--compartment-id ocid1.compartment.oc1..aaaaaaaay______t6q \
--kubernetes-version v1.20.11 \
--name Finance-Node-Pool \
--node-shape VM.Standard2.1 \
--node-defined-tags {"Operations": {"CostCenter": "42"}} \
--node-freeform-tags {"Department": "Finance"}

Using the CLI to Specify Virtual Node Tags

oci ce virtual-node-pool create \
--cluster-id <cluster-ocid> \
--compartment-id <compartment-ocid> \
--display-name <node-pool-name> \
--kubernetes-version <kubernetes-version> \
--placement-configurations "[{\"availabilityDomain\":\"<ad-name>\",\"faultDomain\":[\"FAULT-DOMAIN-<n>\"],\"subnetId\":\"<virtualnode-subnet-ocid>\"}]" \
--nsg-ids "[\"<virtual-node-nsg-ocid>\"]" \
--pod-configuration "{\"subnetId\":\"<pod-subnet-ocid>\",\"nsgIds\":[\"<pod-nsg-ocid>\"],\"shape\":\"<shape-name>\"}" \
--size <number-of-nodes> \
--virtual-node-tags {\"defined-tags\": {<json-name-value-pairs>}, \"freeform-tags\": {<json-name-value-pairs>}}

For example:

oci ce virtual-node-pool create \
--cluster-id ocid1.cluster.oc1.phx.aaaaaaaa______w5q \
--compartment-id ocid1.compartment.oc1..aaaaaaaa______n5q \
--display-name sales-vnp \
--kubernetes-version v1.24.1 \
--placement-configurations "[{\"availabilityDomain\":\"GMvH:PHX-AD-1\",\"faultDomain\":[\"FAULT-DOMAIN-1\"],\"subnetId\":\"ocid1.subnet.oc1.phx.aaaaaaaa______sra\"}]" \
--nsg-ids "[\"ocid1.networksecuritygroup.oc1.phx.aaaaaaaa______hpa\"]" \
--pod-configuration "{\"subnetId\":\"ocid1.subnet.oc1.phx.aaaaaaaa______o7q\",\"nsgIds\":[\"ocid1.networksecuritygroup.oc1.phx.aaaaaaaa______osq\"],\"shape\":\"Pod.Standard.E4.Flex\"}" \
--size 1 \
--virtual-node-tags {\"defined-tags\": {\"Operations\": {\"CostCenter\": \"43\"}}, \"freeform-tags\": {\"Department\": \"Sales\"}}

Using the API

To add and update managed node tags, use the freeformTags and definedTags attributes of the CreateNodePoolNodeConfigDetails and UpdateNodePoolNodeConfigDetails objects used by the CreateNodePool and UpdateNodePool operations.

To add and update virtual node tags, use the freeformTags and definedTags attributes of the CreateVirtualNodePoolDetails.virtualNodeTags and UpdateVirtualNodePoolDetails.virtualNodeTags objects used by the CreateVirtualNodePool and UpdateVirtualNodePool operations.

Applying Tags to Load Balancers

Note

References to load balancer in this section apply to both OCI load balancer resources and OCI network load balancer resources, unless explicitly stated otherwise.

When you create a cluster, you can optionally define tags to apply to load balancer resources created when Kubernetes services of type LoadBalancer are defined. These tags are referred to as initial load balancer tags. You can specify both defined tags and free-form tags as initial load balancer tags.

You can override initial load balancer tags using annotations on the Kubernetes service (see Overriding Initial Load Balancer Tags). If you specify the annotations when defining the Kubernetes service, none of the initial load balancer tags specified in the cluster definition are applied to the load balancer resource. Instead, the tags specified by the annotations are applied to the load balancer resource.

Tag defaults with default values that are specified for the compartment are automatically applied to the load balancer resource as well. Note that Container Engine for Kubernetes does not currently support tag defaults with user-applied values.

Note the following:

  • Tags are only applied to load balancers when the load balancers are first created. If you update the initial load balancer tags, the changes only apply to new load balancers (assuming the tags are not overridden by annotations on the Kubernetes service). Tags applied to existing load balancers are unaffected.
  • If you apply a cost-tracking defined tag to a load balancer resource, you can include load balancer usage in budgets (see Using Cost-Tracking Tags).
  • To apply defined tags from a tag namespace belonging to one compartment to a load balancer resource belonging to a different compartment, you must include a policy statement to allow the cluster to use the tag namespace. See Additional IAM Policy when a Cluster and a Tag Namespace are in Different Compartments.

Using the Console to Specify Initial Load Balancer Tags

To specify an initial load balancer tag to apply to load balancer resources created for a new cluster:

  1. Follow the instructions in Using the Console to create a Cluster with Explicitly Defined Settings in the 'Custom Create' workflow to create a new cluster.
  2. Display the Initial load balancer tags section of the Create Cluster page.
  3. To add a defined tag to load balancer resources:
    • Tag Namespace: Select the tag namespace to which the tag belongs.
    • Tag Key: Select the name of the defined tag to apply to load balancer resources.
    • Tag Value: Either select the value for the tag from a pre-defined list of values, or enter a new value, or leave blank (depending on how the defined tag has been set up).
  4. To add a free-form tag to load balancer resources:
    • Tag Namespace: Set to None (free-form tags do not belong to a tag namespace).
    • Tag Key: Enter a name for the free-form tag to apply to load balancer resources.
    • Tag Value: Enter a value for the tag to apply to load balancer resources.

To update initial load balancer tags to apply to new load balancer resources created for a cluster:

  1. Follow the instructions in Modifying Kubernetes Cluster Properties to update an existing cluster.
  2. Display the Initial Load Balancer Tags tab of the Cluster Details page.
  3. Click Add Tags to add, remove, and change the value of defined tags and free-form tags applied to new load balancer resources.

    Note that tags are only applied to load balancers when the load balancers are first created. So if you update initial load balancer tags, the changes only apply to new load balancer resources. Tags already applied to existing load balancer resources are unaffected.

Using the CLI to Specify Initial Load Balancer Tags

oci ce node-pool create \
--cluster-id <cluster-ocid> \
--compartment-id <compartment-ocid> \
--kubernetes-version <kubernetes-version> \
--name <node-pool-name> \
--node-shape <node-shape>  \
--service-lb-defined-tags <json-name-value-pairs> \
--service-lb-freeform-tags <json-name-value-pairs>

For example:

oci ce node-pool create \
--cluster-id ocid1.cluster.oc1.iad.aaaaaaaaaf______jrd \
--compartment-id ocid1.compartment.oc1..aaaaaaaay______t6q \
--kubernetes-version v1.20.11 \
--name Finance-Node-Pool \
--node-shape VM.Standard2.1 \
--service-lb-defined-tags {"Operations": {"CostCenter": "42"}} \
--service-lb-freeform-tags {"Department": "Finance"}

Using the API to Specify Initial Load Balancer Tags

Use the freeformTags and definedTags attributes of the ServiceLbConfigDetails object used by the CreateCluster and Update Cluster operations to add and update initial load balancer tags.

Overriding Initial Load Balancer Tags

You can override initial load balancer tags specified for a cluster using annotations in the definition of a Kubernetes service of type LoadBalancer, as follows:

  • For load balancer resources:
    • To override defined initial load balancer tags for load balancer resources, add the following annotation in the metadata section of the manifest file:

      oci.oraclecloud.com/initial-defined-tags-override: '{"<tag-namespace>": {"<tag-key>": "<tag-value>"}}'
    • To override free-form initial load balancer tags for load balancer resources, add the following annotation in the metadata section of the manifest file:

      oci.oraclecloud.com/initial-freeform-tags-override: '{"<tag-key>": "<tag-value>"}'
  • For network load balancer resources:
    • To override defined initial load balancer tags for network load balancer resources, add the following annotation in the metadata section of the manifest file:

      oci-network-load-balancer.oraclecloud.com/initial-defined-tags-override: '{"<tag-namespace>": {"<tag-key>": "<tag-value>"}}'
    • To override free-form initial load balancer tags for network load balancer resources, add the following annotation in the metadata section of the manifest file:

      oci-network-load-balancer.oraclecloud.com/initial-freeform-tags-override: '{"<tag-key>": "<tag-value>"}'

For example:


apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/initial-defined-tags-override: '{"Operations": {"CostCenter": "42"}}'
    oci.oraclecloud.com/initial-freeform-tags-override: '{"Department": "Finance"}'
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

Note the following:

  • If you specify the above annotations, none of the initial load balancer tags set for the cluster are applied to the load balancer resource. Only the tags specified by the annotations, along with tag defaults, are applied to the load balancer resource.
  • If you change the annotations in the manifest file, the changes only apply to new load balancer resources. Tags already applied to existing load balancer resources are unaffected. You cannot change the tags applied to an existing load balancer resource by changing the annotations. Instead, you have to create a new Kubernetes service of type LoadBalancer with the annotations for the tags you require.

Which Tags Are Applied to Load Balancers?

This image shows a flowchart that provides a graphical representation of how Container Engine for Kubernetes applies tags to load balancer resources created for Kubernetes services of type LoadBalancer. The same information is provided in the surrounding text.

Applying Tags to Block Volumes

When you create a cluster, you can optionally define tags to apply to block volumes created when persistent volume claims (PVCs) are defined. These tags are referred to as initial block volume tags. You can specify both defined tags and free-form tags as initial block volume tags.

You can override the initial block volume tags by defining a new Kubernetes storage class that includes tag parameters, and then using this storage class to create PVCs (see Overriding Initial Block Volume Tags). If you use a storage class that includes tag parameters, none of the initial block volume tags specified in the cluster definition are applied to block volumes. Instead, the tags set in the storage class are applied to the block volume resources.

Tag defaults with default values that are specified for the compartment are automatically applied to the block volume resources as well. Note that Container Engine for Kubernetes does not currently support tag defaults with user-applied values.

Note the following:

  • Tags are only applied to block volumes when the block volumes are first created. If you update the initial block volume tags applied to block volume resources, the changes only apply to new block volumes (assuming the tags are not overridden by parameters in the storage class definition). Tags applied to existing block volumes are unaffected.
  • If you apply a cost-tracking defined tag to block volumes, you can include block volume usage in budgets (see Using Cost-Tracking Tags).
  • To apply defined tags from a tag namespace belonging to one compartment to a block volume resource belonging to a different compartment, you must include a policy statement to allow the cluster to use the tag namespace. See Additional IAM Policy when a Cluster and a Tag Namespace are in Different Compartments.

Using the Console to Specify Initial Block Volume Tags

To specify an initial block volume tag to apply to block volume resources created for a new cluster:

  1. Follow the instructions in Using the Console to create a Cluster with Explicitly Defined Settings in the 'Custom Create' workflow to create a new cluster.
  2. Display the Initial block volume tags section of the Create Cluster page.
  3. To add a defined tag to block volume resources:
    • Tag Namespace: Select the tag namespace to which the tag belongs.
    • Tag Key: Select the name of the defined tag to apply to block volume resources.
    • Tag Value: Either select the value for the tag from a pre-defined list of values, or enter a new value, or leave blank (depending on how the defined tag has been set up).
  4. To add a free-form tag to block volume resources:
    • Tag Namespace: Set to None (free-form tags do not belong to a tag namespace).
    • Tag Key: Enter a name for the free-form tag to apply to block volume resources.
    • Tag Value: Enter a value for the tag to apply to block volume resources.

To update initial block volume tags to apply to new block volume resources created for a cluster::

  1. Follow the instructions in Modifying Kubernetes Cluster Properties to update an existing cluster.
  2. Display the Initial Block Volume Tags tab of the Cluster Details page.
  3. Click Add Tags to add, remove, and change the value of defined tags and free-form tags applied to new block volume resources.

    Note that tags are only applied to block volume resources when the block volume resources are first created. So if you update initial block volume tags, the changes only apply to new block volume resources. Tags already applied to existing block volume resources are unaffected.

Using the CLI to Specify Initial Block Volume Tags

oci ce node-pool create \
--cluster-id <cluster-ocid> \
--compartment-id <compartment-ocid> \
--kubernetes-version <kubernetes-version> \
--name <node-pool-name> \
--node-shape <node-shape>  \
--persistent-volume-defined-tags <json-name-value-pairs> \
--persistent-volume-freeform-tags <json-name-value-pairs>

For example:

oci ce node-pool create \
--cluster-id ocid1.cluster.oc1.iad.aaaaaaaaaf______jrd \
--compartment-id ocid1.compartment.oc1..aaaaaaaay______t6q \
--kubernetes-version v1.20.11 \
--name Finance-Node-Pool \
--node-shape VM.Standard2.1 \
--persistent-volume-defined-tags {"Operations": {"CostCenter": "42"}} \
--persistent-volume-freeform-tags {"Department": "Finance"}

Using the API to Specify Initial Block Volume Tags

Use the freeformTags and definedTags attributes of the PersistentVolumeConfigDetails object used by the CreateCluster and Update Cluster operations to add and update initial block volume tags.

Overriding Initial Block Volume Tags

You can override initial block volume tags applied to a block volume resource using parameters in a storage class manifest file as follows:

  • To override defined initial block volume tags, include the following parameter in the parameters section of the StorageClass definition:

    oci.oraclecloud.com/initial-defined-tags-override: '{"<tag-namespace>": {"<tag-key>": "<tag-value>"}}'
  • To override free-form initial block volume tags, include the following parameter in the parameters section of the StorageClass definition:

    oci.oraclecloud.com/initial-freeform-tags-override: '{"<tag-key>": "<tag-value>"}'

For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: oci-bv-specifictags
provisioner: blockvolume.csi.oraclecloud.com
parameters:
  attachment-type: "paravirtualized"
  oci.oraclecloud.com/initial-freeform-tags-override: '{"Department": "Finance"}'
  oci.oraclecloud.com/initial-defined-tags-override: '{"Operations": {"CostCenter": "42"}}'
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Note the following:

  • If you specify the above parameters, none of the initial block volume tags set for the cluster are applied to the block volume resource. Only the tags specified by the parameters, along with tag defaults, are applied to the block volume resource.
  • Having applied a manifest file containing a StorageClass definition to create the storage class, you cannot subsequently change the tag parameter values by updating the definition and reapplying the manifest file. Once a storage class has been created, it is immutable. To specify different tag parameter values to the ones you previously specified, you have to create a new StorageClass definition that includes the new values for the tag parameters. Then specify the new storage class in the PVC definition.

Which Tags Are Applied to Block Volumes?

This image shows a flowchart that provides a graphical representation of how Container Engine for Kubernetes applies tags to block volume resources. The same information is provided in the surrounding text.

Managing Access By Applying Tags to Cluster-Related Resources

Having applied tags to cluster-related resources, you can control access to other resources by including the tags in IAM policies.

Including tags in IAM policies is particularly useful when authorizing worker nodes to access other resources. If you specify node tags when initially defining a node pool and the node pool is subsequently scaled out, the node tags are automatically applied to any new worker nodes that are created. Provided you have included the node tags in an IAM policy that grants access to other resources, the new worker nodes are automatically authorized to access the same resources as existing worker nodes in the node pool.

Note that you cannot use free-form tags in IAM policies to control access to resources.

Example: Authorizing a managed node pool to read object storage objects

To authorize only worker nodes in a particular managed node pool to read object storage objects in a certain compartment:

  1. Define a new managed node pool and specify:
    • A name for the node pool that is unique in the compartment. For example, acme-pool-one
    • A node tag set to the name of the node pool. For example, a defined tag named acme-nodepool-name in a tag namespace named acme-project, set to the value acme-pool-one.

    When you create the node pool, the node tag acme-project.acme-nodepool-name='acme-pool-one' is applied to all the compute instances hosting worker nodes in the node pool.

  2. Define a dynamic group to represent the node pool:
    1. Open the navigation menu and click Identity & Security. Under Identity, click Dynamic Groups.
    2. Follow the instructions in To create a dynamic group, and give the dynamic group a name (for example, acme-oke-pool-one-dyn-grp).
    3. Enter a rule that includes all worker nodes in the compartment to which the node tag has been applied, in the format:

      All {instance.compartment.id = '<compartment_ocid>', tag.<tagnamespace>.<tagkey>.value = '<tagvalue>'}

      For example:

      All {instance.compartment.id = 'ocid1.compartment.oc1..aaaaaaaa23______smwa', tag.acme-project.acme-nodepool-name.value = 'acme-pool-one'}
    4. Click Create Dynamic Group.
  3. Define an IAM policy that authorizes the dynamic group to read object storage objects in a compartment:
    1. Open the navigation menu and click Identity & Security. Under Identity, click Policies.
    2. Follow the instructions in To create a policy, and give the policy a name (for example, acme-oke-read-object-dyn-grp-policy).
    3. Enter a policy statement to allow the dynamic group to read object storage objects, in the format:

      Allow dynamic-group <dynamic-group-name> to read objects in compartment <compartment-name>

      where:

      • <dynamic-group-name> is the name of the dynamic group you created earlier.
      • <compartment-name> is the name of the compartment containing the object storage objects.

      For example:

      Allow dynamic-group acme-oke-pool-one-dyn-grp to read objects in compartment acme-oke-obj-storage-comp
    4. Click Create to create the new policy.

    All worker nodes in the node pool can now read object storage objects in the compartment you specified.

Retired Tags and Cluster-Related Resources

Oracle Cloud Infrastructure Tagging supports the concept of retiring tags (see Tags and Tag Namespace Concepts). If a tag is retired, the tag is not removed from resources it was previously applied to. However, the retired tag can no longer be applied to new resources.

In the context of Kubernetes clusters, existing cluster-related resources to which a tag has already been applied continue to function as normal if the tag is subsequently retired. The tag still exists as metadata on those resources, and you can still use the retired tag in operations such as listing, sorting, and reporting.

However, attempting to apply a retired tag to new cluster-related resources will return an error. If a tag is retired that was previously specified as a tag default, node tag, initial load balancer tag, or initial block volume tag, then new corresponding cluster-related resources cannot be created.

In particular, if a managed node pool definition specifies a node tag and the tag is subsequently retired, the node pool cannot scale out because new worker nodes cannot be created. To enable the node pool to scale out, remove the retired tag from the node pool definition.

Additional IAM Policy when a Cluster and a Tag Namespace are in Different Compartments

To apply defined tags from a tag namespace belonging to one compartment to cluster-related resources belonging to a different compartment, you must include a policy statement similar to the following in an IAM policy:

Allow any-user to use tag-namespace in tenancy where all {request.principal.type = ‘cluster’}

If you consider this policy statement to be too permissive, you can restrict the permissions to explicitly specify the compartment to which the tag namespace belongs, and/or to explicitly specify the cluster that belongs to a different compartment. For example:

Allow any-user to use tag-namespace in compartment <compartment-ocid> where all { request.principal.id = '<cluster-ocid>' }