Health Checks & Failover

📘 Chapter 6: Load Balancing & Reverse Proxies ⏱️ 9 min read 🏗️ Lesson 026

A load balancer is only useful if it knows which servers are alive. Health checks are the mechanism — the LB periodically probes each backend to determine if it can handle traffic. When a server fails, failover kicks in to route around the damage automatically.

Why Health Checks Matter

Without health checks, a load balancer blindly sends requests to dead servers. Users get connection timeouts, 502 errors, and a terrible experience. Health checks detect problems before users hit them.

Active vs Passive Health Checks

Active Health Checks

The LB periodically sends probe requests (every 5-30 seconds) to each backend:

  • TCP check: Can I open a connection to port 8080?
  • HTTP check: Does GET /health return status 200?
  • Custom check: Does the response include {"db": "ok", "cache": "ok"}?

If a server fails N consecutive checks (e.g., 3), it's marked unhealthy and removed from the pool.

Passive Health Checks

The LB observes real traffic responses — no extra probe requests needed:

  • Count 5xx errors: if a server returns 5 errors in 30 seconds → mark unhealthy
  • Track timeouts: if responses take >10s consistently → mark degraded
  • Monitor connection resets: TCP RST from backend → something is wrong

Passive checks are faster to detect issues (no waiting for next probe interval) but only work if the server is receiving traffic.

Health Check States & LB Routing LB Server A ✓ ✓ Server B ⚠️ ! Server C ✗ ✗ no traffic Health States Healthy — receives full traffic Degraded — reduced traffic Failed — removed from pool Recovery: pass 2+ checks → back in pool Active probes every 10s • 3 consecutive failures = unhealthy • 2 passes = healthy again
Figure 1: The LB routes traffic only to healthy servers. Failed servers receive no traffic until they recover.

Liveness vs Readiness Probes

Kubernetes Concepts That Apply Broadly

  • Liveness probe: "Is this process alive?" If it fails, restart the container. Detects deadlocks, infinite loops, corrupted state.
  • Readiness probe: "Can this instance handle traffic right now?" If it fails, remove from the LB pool but don't restart. Detects startup in progress, dependency unavailable, overloaded.

The distinction matters: a server starting up is alive (don't restart it) but not ready (don't send it traffic yet).

What to Check

Health Endpoint Design

A good /health endpoint checks what matters:

  • Shallow check: GET /health → 200 OK (process is up and listening)
  • Deep check: GET /health/ready → verifies DB connectivity, cache reachable, disk space OK

Caution: Deep checks can backfire. If your DB has a momentary hiccup, all servers might simultaneously fail their health check, causing a total outage. Consider: should a DB issue mark the server unhealthy, or should the server return degraded responses?

Failover Strategies

Active-Passive Failover Sequence 1. Normal Primary ✓ Standby 💤 All traffic → Primary → 2. Failure Primary ✗ Standby ⚡ Health check fails! → 3. Promote Primary (dead) New Primary ✓ Traffic → Standby → 4. Recovered Old Primary 🔧 Serving ✓ Old primary repairs/becomes standby t=0 t=30s (detected) t=31s (failover) t=5min (repaired) Active-Active alternative: Both servers handle traffic simultaneously. If one dies, the other absorbs 100% — no promotion needed, but each must handle 2x load.
Figure 2: Active-passive failover: standby promotes when primary fails. Active-active: both serve, survivor takes full load.

DNS-Based vs LB-Based Failover

  • LB-based: Instant. LB detects failure in seconds and stops routing. No client changes needed. Best for services behind a load balancer.
  • DNS-based: Slow (minutes). Change DNS record to point to standby. Clients cache old DNS (TTL). Used for region-level failover or when LB itself fails.

Lesson: Use LB-based failover for individual servers. Reserve DNS failover for catastrophic (entire datacenter) failures.

Cascading Failures: When Health Checks Go Wrong

The Thundering Herd Problem

Scenario: 5 servers behind a LB. Server A goes down. The LB redistributes A's traffic across B, C, D, E. But now each handles 25% more load. If they were at 80% capacity, they're now at 100%. Server B buckles, fails its health check. Now C, D, E handle even more... cascade failure.

Mitigations:

  • Keep servers at <60% capacity (headroom for failover absorption)
  • Use gradual ramp-up (don't send full traffic to a recovering server instantly)
  • Circuit breakers: stop retrying a failing dependency
  • Shed load intentionally (return 503) rather than collapsing entirely

🏢 Real-World: GitHub's Graceful Deploy Failover

When GitHub deploys new code, they use health checks as a coordination mechanism:

  1. Drain: Server signals "not ready" → LB stops sending new requests
  2. Wait: Existing requests finish (graceful shutdown period)
  3. Deploy: New code is deployed and server starts up
  4. Warm: Server initializes caches, loads config
  5. Ready: Health check passes → LB routes traffic back

Users never see errors because the LB routes around servers being deployed. This "rolling deploy" pattern enables GitHub to deploy dozens of times per day with zero downtime.

Interactive: Kill Servers and Watch Failover

Click servers to "kill" them and watch the LB detect failure and reroute traffic:

Load Balancer
⚖️
Server 1
● Healthy
Load: 25%
Server 2
● Healthy
Load: 25%
Server 3
● Healthy
Load: 25%
Server 4
● Healthy
Load: 25%
💡 Click any server to simulate a failure. Click again to bring it back online.