Health Check Policies for Network Load Balancers

Using Health Status

At the highest level, network load balancer health reflects the health of its components. The health status indicators provide information you might need to drill down and investigate an existing issue. Some common issues that the health status indicators can help you detect and correct include:

A health check is misconfigured.

In this case, all the backend servers for one or more of the affected listeners report as unhealthy. If your investigation finds that the backend servers don't have problems, then a backend set probably includes a misconfigured health check.

A listener is misconfigured.

All the backend server health status indicators report OK, but the network load balancer doesn't pass traffic on a listener.

The listener might be configured to:

• Listen on the wrong port.

• Use the wrong protocol.

• Use the wrong policy.

If your investigation shows that the listener isn't at fault, check the security list configuration.

A security rule is misconfigured.

Health status indicators help you diagnose two cases of misconfigured security rules:

• All entity health status indicators report OK, but traffic doesn't flow (as with misconfigured listeners). If the listener isn't at fault, check the security rule configuration.

• All entity health statuses report as unhealthy. You have checked the health check configuration and the services run correctly on the backend servers.

  In this case, the security rules might not include the IP range for the source of the health check requests. You can find the health check source IP on the Details page for each backend server. You can also use the API to find the IP in the sourceIpAddress field of the HealthCheckResult object.

  Note
  
  

  The source IP for health check requests comes from a compute instance managed by the Network Load Balancer service.

One or more of the backend servers reports as unhealthy.

A backend server might be unhealthy or the health check might be misconfigured. To see the corresponding error code, check the status field on the backend server's Details page. You can also use the API to find the error code in the healthCheckStatus field of the HealthCheckResult object.

Other cases in which health status might prove helpful include:

• VCN network security groups or network security lists block traffic.

• Compute instances have misconfigured route tables.

Health status is updated every three minutes. No finer granularity is available.

Health status doesn't provide historical health data.

Health Check Best Practices

Configure the health check protocol to match the application or service. If you run an HTTP service, be sure to configure an HTTP-level health check. If you run a TCP-level health check against an HTTP service, you might not get an correct response. The TCP handshake can succeed and indicate that the service is up even when the HTTP service is incorrectly configured or having other issues. Although the health check appears good, customers might experience transaction failures.

For example:

• The backend HTTP service has issues when talking to the health check URL and the health check URL returns 5XX messages. An HTTP health check catches the message from the health check URL and marks the service as down. In this case, a TCP health check handshake succeeds and marks the service as healthy, even though the HTTP service might not be usable.

• The backend HTTP service responds with 4XX messages because of authorization issues or no configured content. A TCP health check doesn't catch these errors.

Common Side Effects of Health Check Misconfiguration

The following are common side effects of health check misconfiguration, and can be used to troubleshoot issues.

• Wrong port

  In this scenario, all the backend servers report as unhealthy. If the backend servers don't have any problems, you might have made a mistake setting the port. The port must be listening and has allowed traffic on the backend.

  OCI Logging Error: errno":"EHOSTUNREACH","syscall":"connect".

• Wrong patch

  In this scenario, this scenario, all the backend servers report as unhealthy. If the backend servers don't have any problems, you might have made a mistake setting the path for the HTTP health check it needs to match an actual application on the backend. In you can use a curl test from a system in the same network.

  For example:

  $ curl -i http://10.0.0.5/health

  You receive the configured status code in the response OCI Logging Error:

  "msg":"invalid statusCode","statusCode":404,"expected":"200".

• Wrong protocol

  In this scenario, all the backend servers report as unhealthy. If the backend servers don't have any problems, you might have made a mistake setting the protocol it needs to match the protocol that's listening on the backend. For example: We only support TCP and HTTP health checks. If the backend server is using HTTPS, use TCP as the protocol.

  OCI Logging Error:

  "code":"EPROTO","errno":"EPROTO".

• Wrong status code

  In this scenario, all the backend servers report as unhealthy. If the backend servers don't have any problems, for an HTTP health check you might have made a mistake setting the status code to match the actual status code being returned from the backend. A common scenario is when a backend is returning a 302 and you're expecting a 200. This result is likely the backend sending you to a sign in page or another location on the server. In this scenario, you can either fix the backend to return the expected code or use 302 in the health check configuration.

  OCI Logging Error:

  "msg":"invalid statusCode","statusCode":XX,"expected":"200" 

  where XX to be the status code that's returned.

• Wrong regex pattern

  All the backend servers report as unhealthy. If the backend servers don't have any problems, you might have made a mistake setting an incorrect regex pattern consistent with the body, or the backend isn't returning the expected body. In this scenario, you can either change the backend to match the pattern or correct the pattern to match the backend. The following are some specific pattern examples.

  • Any Content - ".*"

  • A page returning the value "Status:OK:" - "Status:OK:.*"

  • OCI Logging Error: "response match result: failed"

• Misconfigured Network Security Groups, security lists, or local firewall

All or some backend servers report as unhealthy. If the backend servers don't have any problems, you might have made a mistake configuring either the NSGs, Security Lists, or local firewalls such as firewalld, iptables, or SELiinux. In this scenario you can use a curl or netcat test from a system that belongs to the same subnet and NSG as the balancer instance HTTP:

For example:

$ curl -i http://10.0.0.5/health TCP: ex: nc -zvw3 10.0.05 443.

You can check the local firewall by using the following command:

firewall-cmd --list-all --zone=public.

If the firewall is missing the expected rules you can use a command set such as this to add the service: (this example is for HTTP port 80):

• firewall-cmd --zone=public --add-service=http

• firewall-cmd --zone=public --permanent --add-service=http