Load Balancer Health Checks: Endpoints, Thresholds and Connection Draining

A load balancer is only as good as its idea of which servers are healthy. If health checks are too shallow, traffic keeps flowing to a server that returns errors. If they are too sensitive or too deep, a brief database hiccup marks every server unhealthy at once and takes down a site that was mostly working. Good health checks remove a failing backend quickly, return it to service only when it is ready, and never turn a partial problem into a total outage. This guide covers how to design them.

The concepts apply to any load balancer: a cloud load balancer, HAProxy, nginx, or a Kubernetes service. Setting names differ, so check your product's documentation for exact options.

Active vs passive checks

• Active checks: the load balancer itself sends a probe on a schedule (a TCP connect or an HTTP request) and marks the backend healthy or unhealthy by the result.
• Passive checks: the load balancer watches real traffic and marks a backend unhealthy after a number of failed real requests. nginx open source works this way using max_fails and fail_timeout on upstream servers; active HTTP checks in nginx are a commercial feature, so verify what your edition supports.

Best practice is to use active checks as the main signal and, where available, passive detection on top of them, so a backend that passes a probe but fails real requests is still caught.

Choose the check level

TCP checks are a fallback for non-HTTP services. For web backends use an HTTP check. The Port Checker is the quickest way to test the TCP layer from outside, and the HTTP Status Checker tests the HTTP layer.

Design the health endpoint

Create a dedicated endpoint such as /healthz instead of probing the homepage. The homepage is heavy, may be cached, redirects to HTTPS, or depends on cookies, all of which complicate checks.

A good health endpoint:

• Returns 200 when the instance can serve traffic and a 5xx (commonly 503) when it can't.
• Is fast and cheap: no heavy queries.
• Bypasses authentication, redirects, and caching so the answer reflects the instance itself. Make sure a CDN or cache doesn't answer for it.
• Is accessible only from the load balancer or internal networks, not the public internet, if it exposes internals.
• Returns small, stable output.

Liveness vs readiness

Borrowing language from container orchestration, keep two ideas separate:

• Liveness: is the process alive and not wedged? If not, restart it.
• Readiness: can it serve requests right now? If not, take it out of rotation but don't restart it.

A readiness endpoint may check that the application has started, caches are warm, and a database connection can be opened. A liveness endpoint should be much more minimal.

Be careful with dependency checks

If every backend's health check queries the shared database, then a database blip makes every backend report unhealthy simultaneously, and the load balancer has no healthy servers left. The site is down even though the web tier is fine and pages that don't need the database could have been served.

Safer patterns:

• Check local health (process, disk space, ability to run) in the endpoint used by the load balancer.
• Check shared dependencies in a separate endpoint used for monitoring and alerts, not for pulling servers.
• If you do include a dependency, use a short timeout and consider it failing only after repeated errors.
• Configure a fail-open behavior where your load balancer supports it, so that if all backends appear unhealthy, traffic still goes somewhere. Look for options such as a "panic" or "fail open" mode in your product's docs.

A minimal example in Node:

app.get('/healthz', (req, res) => {
  if (shuttingDown) return res.status(503).send('draining');
  res.status(200).send('ok');
});

And in nginx, serving a static response without touching the app:

location = /healthz {
    access_log off;
    default_type text/plain;
    return 200 "ok\n";
}

That nginx version only proves nginx is alive, not the application behind it. Proxy to the app's own health route when you need the app's status.

Intervals, timeouts and thresholds

Four values define behavior:

• Interval: time between probes.
• Timeout: how long to wait for each response. It must be shorter than the interval and long enough to survive normal latency.
• Unhealthy threshold (fall): consecutive failures before removal.
• Healthy threshold (rise): consecutive successes before re-adding.

The time to detect a failure is approximately interval x unhealthy threshold. A 5-second interval and 3 failures detects an outage in about 15 seconds. That is not the same as zero impact: during those seconds some requests will fail, which is why retries (below) matter.

Tuning guidance:

• Use a higher healthy threshold than unhealthy threshold or similar, so a flapping server isn't returned to rotation after one lucky success. Requiring a few consecutive successes protects users from half-recovered instances.
• Keep the timeout comfortably above your endpoint's normal response time. A timeout of 1 second on a check that sometimes takes 900 ms will flap.
• Don't set intervals so short that checks become meaningful load on small backends, especially when many load balancer nodes each probe independently.
• Include a slow start or warm-up period, if supported, so a new or recovered instance gets traffic gradually while caches warm.

HAProxy example

backend web
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    default-server inter 5s fall 3 rise 2 slowstart 30s
    server web1 10.0.0.11:8080 check
    server web2 10.0.0.12:8080 check
    server web3 10.0.0.13:8080 check backup

Here each server is probed every 5 seconds, marked down after 3 failures, and marked up after 2 successes. backup servers only receive traffic when the primary servers are down. Validate with haproxy -c -f /etc/haproxy/haproxy.cfg before reloading.

nginx open source passive example

upstream app {
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
}

server {
    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
    }
}

After 3 failures within the window, the server is considered unavailable for fail_timeout. proxy_next_upstream retries a failed request on another server. Be careful about retrying non-idempotent requests like POST: nginx does not retry them by default for good reason. For details on timeouts, see nginx 502 and 504 timeout tuning.

Connection draining

When you remove a server for a deploy or scale-in, killing it abruptly breaks in-flight requests. Draining (also called connection draining or deregistration delay, depending on the product) stops sending new requests to the server while letting existing ones finish.

A graceful removal sequence:

1. Mark the backend as draining in the load balancer so no new connections arrive.
2. Wait for in-flight requests to complete, up to a maximum drain time longer than your slowest normal request.
3. Stop or restart the application.
4. Start it, wait until the health endpoint passes, and let the load balancer add it back.

HAProxy lets you do this at runtime through the stats socket:

echo "set server web/web1 state drain" | socat stdio /var/run/haproxy.sock
# wait for sessions to finish, then deploy
echo "set server web/web1 state ready" | socat stdio /var/run/haproxy.sock

Make the application cooperate: when it receives a shutdown signal (SIGTERM), it should start failing its readiness check, finish active requests, then exit. The Node snippet above uses a shuttingDown flag for this. Long-lived connections like WebSockets need a longer drain time or a client reconnect strategy.

Common failure patterns

• Flapping: servers repeatedly marked up and down. Cause: timeout too tight, thresholds of 1, or a health check that runs a slow query. Fix: widen timeout, raise thresholds, simplify the check.
• All backends down at once: shared dependency in the health check. Separate local and dependency health.
• Healthy but broken: TCP check passes while the app returns 500s. Use an HTTP check, and add passive failure detection.
• Health check redirected: probe to /healthz gets 301 to HTTPS and is treated as failure. Probe the right scheme and port, and exempt the path from redirects.
• Unhealthy after deploy: application takes longer to start than the threshold allows. Add a startup grace period.
• Check blocked by firewall or host header: probes come from load balancer IPs; the application or WAF may reject them. Allow them and set the Host header the app expects.

Monitor the load balancer from outside

The load balancer's view of health is internal. Add an external check that sees what users see. The Uptime Monitor probes your public URL from outside every 5 minutes, and can check for expected keywords or JSON values, which catches partial failures a status code misses. Pair it with a public status page and read how to reduce false alerts in monitoring to apply the same threshold thinking to your alerts. The related alert fatigue and on-call basics covers who gets paged.

Next steps

• Probe your public endpoints with the HTTP Status Checker and Port Checker.
• Set up outside-in checks with the Uptime Monitor.
• Tune the proxy layer with nginx 502 and 504 timeout tuning.
• Understand 502 and 504 responses in fix 502 bad gateway and fix 504 gateway timeout.

FAQ

How often should a load balancer check health?

Common intervals are a few seconds to tens of seconds. Pick an interval and failure threshold so detection time (interval times failures) is acceptable, and the check load is small compared with real traffic.

Should the health check test the database?

Usually not in the check that removes servers from rotation. A shared database failure would mark every backend unhealthy at once. Monitor dependencies separately and alert on them.

What does connection draining do?

It stops sending new requests to a backend while letting in-flight requests finish, so deploys and removals don't cut off users mid-request.

Why does my server flap between healthy and unhealthy?

Usually the check timeout is too short, thresholds are too low, or the endpoint does slow work. Increase the timeout, require several consecutive successes, and keep the endpoint light.

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.