Provisioning OCI Network Load Balancers for Kubernetes Services of Type LoadBalancer

To provision a network load balancer for a Kubernetes service of type LoadBalancer, add the following annotation in the metadata section of the manifest file:

oci.oraclecloud.com/load-balancer-type: "nlb"

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

Note that lb is the default value of the oci.oraclecloud.com/load-balancer-type annotation. If you do not explicitly include the annotation in the service definition, the default value of the annotation is used and a load balancer (rather than a network load balancer) is provisioned.

When provisioning a network load balancer for a Kubernetes service of type LoadBalancer, you can specify that requests terminate at the client IP address specified in the headers of IP packets, rather than being proxied to other worker nodes in the cluster.

By default, requests are proxied to other worker nodes in the cluster.

Specifying that requests terminate at the client IP address (rather than being proxied) can improve performance in very large clusters with thousands of worker nodes by eliminating traffic between worker nodes. Specifying that requests terminate at the client IP address can also simplify implementation and remove potential security concerns by enabling you to set up security rules (in a network security group (recommended) and/or a security list) for the worker nodes in the cluster that only allow ingress traffic from the network load balancer's CIDR block.

To terminate requests at the client IP address, add the following setting in the spec section of the manifest file:

externalTrafficPolicy: Local

To proxy requests to other worker nodes in the cluster, add the following setting in the spec section of the manifest file:

externalTrafficPolicy: Cluster

Note that Cluster is the default value of the externalTrafficPolicy setting. If you do not explicitly include the setting in the service definition, the default value of the setting is used.

Also note that if externalTrafficPolicy is set to Cluster, client IP addresses are not preserved regardless of the value of the oci-network-load-balancer.oraclecloud.com/is-preserve-source annotation. Requests fail with an error if externalTrafficPolicy is set to Cluster and the oci-network-load-balancer.oraclecloud.com/is-preserve-source annotation is explicitly set to either true or false. See Preserving The Client IP Address.

To terminate requests at the client IP address, you must also have set up the following security rules:

• You must have set up a security rule (in a network security group (recommended) and/or a security list) for the worker nodes in the cluster to allow ingress traffic from the CIDR block where the client connections are made, to all node ports (30000 to 32767). If the application is exposed to the Internet, set the security rule's Source CIDR block to 0.0.0.0/0. Alternatively, set the security rule's Source CIDR block to a specific CIDR block (for example, if the client connections come from a specific subnet).

  State	Source	Protocol/Dest. Port	Description
  Stateful	0.0.0.0/0 or subnet CIDR	ALL/30000-32767	Allow worker nodes to receive connections through OCI Network Load Balancer.

• You must have set up the ingress and egress security rules for the network load balancer, as described in Security Rules for Load Balancers and Network Load Balancers.

For example, here is a Kubernetes service definition to terminate requests at the client IP address (rather than being proxied):

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "nlb"
    oci-network-load-balancer.oraclecloud.com/oci-network-security-groups: "ocid1.networksecuritygroup.oc1.phx.aaaaaa....vdfw"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
  - port: 80
  selector:
    app: nginx

When provisioning a network load balancer for a Kubernetes service of type LoadBalancer, you can specify whether to preserve, or prevent the preservation of, the client IP address in the headers of IP packets.

You only have the option to preserve client IP addresses when requests are terminated at the client IP addresses specified in the IP packet headers. That is, when the externalTrafficPolicy setting is set to Local. If externalTrafficPolicy is set to Cluster, client IP addresses are not preserved. See Terminating Requests at the Receiving Node.

To prevent the preservation of client IP addresses, add the following annotation in the metadata section of the manifest file:

oci-network-load-balancer.oraclecloud.com/is-preserve-source: "false"

To preserve the client IP address, add the following annotation in the metadata section of the manifest file:

oci-network-load-balancer.oraclecloud.com/is-preserve-source: "true"

Note that true is the default value of the oci-network-load-balancer.oraclecloud.com/is-preserve-source annotation. If you do not explicitly include the annotation in the service definition, the default value of the annotation is used.

Also note that if externalTrafficPolicy is set to Cluster, client IP addresses are not preserved regardless of the value of the oci-network-load-balancer.oraclecloud.com/is-preserve-source annotation. Requests fail with an error if externalTrafficPolicy is set to Cluster and the oci-network-load-balancer.oraclecloud.com/is-preserve-source annotation is explicitly set to either true or false. Therefore do not add the oci-network-load-balancer.oraclecloud.com/is-preserve-source annotation if externalTrafficPolicy is set to Cluster.

You can preserve client IP addresses when using managed node pools, but not when using virtual node pools.

To preserve the client IP address, you must also have set up the following security rules:

• You must have set up a security rule (in a network security group (recommended) and/or a security list) for the worker nodes in the cluster to allow ingress traffic from the CIDR block where the client connections are made, to all node ports (30000 to 32767). If the application is exposed to the Internet, set the security rule's Source CIDR block to 0.0.0.0/0. Alternatively, set the security rule's Source CIDR block to a specific CIDR block (for example, if the client connections come from a specific subnet).

  State	Source	Protocol/Dest. Port	Description
  Stateful	0.0.0.0/0 or subnet CIDR	ALL/30000-32767	Allow worker nodes to receive connections through OCI Network Load Balancer.

• You must have set up the ingress and egress security rules for the network load balancer, as described in Security Rules for Load Balancers and Network Load Balancers.

For example, here is a Kubernetes service definition that prevents the preservation of the client IP address:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "nlb"
    oci-network-load-balancer.oraclecloud.com/oci-network-security-groups: "ocid1.networksecuritygroup.oc1.phx.aaaaaa....vdfw"
    oci-network-load-balancer.oraclecloud.com/is-preserve-source: "false"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
  - port: 80
  selector:
    app: nginx

When Container Engine for Kubernetes provisions a network load balancer for a Kubernetes service of type LoadBalancer, you can define the type of traffic accepted by the listener by specifying the protocol on which the listener accepts connection requests.

Note that if you don't explicitly specify a protocol, "TCP" is used as the default value.

To explicitly specify the listener protocol when Container Engine for Kubernetes provisions a network load balancer for a Kubernetes service of type LoadBalancer, add the following setting in the spec section of the manifest file:

protocol: <value>

where <value> is the protocol that defines the type of traffic accepted by the listener. For example, "UDP". Valid protocols include "UDP" and "TCP".

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  ports:
  - port: 80
    protocol: UDP
  selector:
    app: nginx

When Container Engine for Kubernetes provisions a network load balancer for a Kubernetes service of type LoadBalancer, you can define a policy for the backend set to specify how to distribute incoming traffic to the backend servers. For more information, see Network Load Balancer Policies.

Note that if you don't explicitly specify a policy for the backend set, "FIVE_TUPLE" is used as the default value.

To specify a policy for the backend set when Container Engine for Kubernetes provisions a network load balancer for a Kubernetes service of type LoadBalancer, add the following annotation in the metadata section of the manifest file:

oci-network-load-balancer.oraclecloud.com/backend-policy: <value>

where <value> is one of:

• "TWO_TUPLE": Routes incoming traffic based on 2-Tuple (source IP, destination IP) Hash.
• "THREE_TUPLE": Routes incoming traffic based on 3-Tuple (source IP, destination IP, protocol) Hash.
• "FIVE_TUPLE": Routes incoming traffic based on 5-Tuple (source IP and port, destination IP and port, protocol) Hash.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "nlb"
    oci-network-load-balancer.oraclecloud.com/backend-policy: "THREE_TUPLE"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

You can use the security list management feature to configure how security list rules are managed for an Oracle Cloud Infrastructure network load balancer that Container Engine for Kubernetes provisions for a Kubernetes service of type LoadBalancer. This feature is useful if you are new to Kubernetes, or for basic deployments.

To specify how the Kubernetes security list management feature manages security lists when Container Engine for Kubernetes provisions a network load balancer for a Kubernetes service of type LoadBalancer, add the following annotation in the metadata section of the manifest file:

oci-network-load-balancer.oraclecloud.com/security-list-management-mode: <value>

where <value> is one of:

• "None": (default, and recommended) No security list management is enabled. You have to set up a security rule that allows inbound traffic to the appropriate ports for node port ranges, the kube-proxy health port, and the health check port ranges. Additionally, you have to set up security rules to allow inbound traffic to network load balancers (see Security Rules for Load Balancers and Network Load Balancers).
• "All": All required security list rules for network load balancer services are managed.
• "Frontend": Only security list rules for ingress to network load balancer services are managed. You have to set up a security rule that allows inbound traffic to the appropriate ports for node port ranges, the kube-proxy health port, and the health check port ranges.

Oracle recommends that you explicitly set oci-network-load-balancer.oraclecloud.com/security-list-management-mode to None.

In clusters with managed nodes, if you don't explicitly specify a management mode, security list management is not enabled (equivalent to "None"). In clusters with virtual nodes, security list management is never enabled and you always have to manually configure security rules (equivalent to "None").

Note that there are limits to the number of ingress and egress rules that are allowed in a security list (see Security List Limits). If the number of ingress or egress rules exceeds the limit, and <value> is set to "All" or "Frontend", creating or updating the load balancer fails.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "nlb"
    oci-network-load-balancer.oraclecloud.com/security-list-management-mode: "Frontend"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx