Chapter 6 Interview: Load Balancing

Alright so for a high-traffic API, I'd think about this in tiers. You don't just slap one load balancer in front and call it a day — that becomes your single point of failure.

First, DNS-level load balancing to distribute across multiple data centers or availability zones. Then within each zone, an L4 load balancer — something like AWS NLB or HAProxy in TCP mode — that handles the raw connection routing with minimal overhead. Behind that, you might have L7 load balancers (like Nginx or Envoy) that can do content-based routing, like sending /api/v2 to a different backend pool than /api/v1.

For the algorithm, I'd start with least-connections for most API workloads. Round-robin works if all requests are roughly equal cost, but APIs often have endpoints that vary wildly in processing time. Least-connections naturally adapts to that.

Health checks are non-negotiable — both shallow (TCP port open?) and deep (can the service actually process a request?). Unhealthy backends get pulled from the pool within seconds. And I'd set up the LB pair in active-passive or active-active for redundancy.

• Multi-tier: DNS → L4 (connection) → L7 (content-aware) routing
• Algorithm: least-connections for variable-cost endpoints
• Health checks: shallow (port) + deep (functional) with fast failure detection
• Redundancy: LB itself must not be a SPOF — active-passive or active-active pair

So L4 and L7 refer to the OSI model layers — transport and application. The key difference is how much the load balancer understands about the traffic it's routing.

L4 operates at the TCP/UDP level. It sees source IP, destination IP, and port numbers. That's it. It makes routing decisions based on those — maybe simple round-robin or least-connections. It's fast because it doesn't need to inspect the payload. Think of it as a traffic cop directing cars without knowing what's inside them.

L7 actually terminates the HTTP connection and inspects the request — the URL path, headers, cookies, even the body sometimes. This lets you do smart routing: send mobile traffic to one pool, route by API version, do A/B testing by header, sticky sessions by cookie. Way more flexible but more compute overhead because it's parsing every request.

In practice I'd use L4 as the front door for raw throughput and connection distribution, then L7 behind it for intelligent routing decisions. L4 when you need speed and scale, L7 when you need smarts.

• L4: TCP/UDP level — IP + port only, fast, no content inspection
• L7: HTTP level — sees URLs, headers, cookies; enables content-based routing
• Performance: L4 handles more connections/sec due to less processing
• Use together: L4 at the edge for scale, L7 internally for routing logic

The goal is that when a backend dies, users don't notice. Zero visible impact. Here's how I'd set that up.

First, active health checks running every few seconds. The load balancer pings each backend — if it fails N consecutive checks (usually 2-3), that server gets pulled from the rotation. No more traffic goes there. This handles gradual failures.

For sudden crashes — a server dies mid-request — you need connection-level detection. The LB sees the TCP connection drop or a 502/503 response and immediately retries that request on a different backend. The user sees maybe 50ms extra latency instead of an error. Key thing: only retry idempotent requests automatically. You don't want to retry a payment POST.

Then there's graceful shutdown — when you're deploying new code, the server signals it's draining. The LB stops sending new connections but lets existing ones finish. Connection draining with a timeout, then force-kill. Zero-downtime deployments.

And you need enough headroom — if you're running 4 servers at 80% capacity and one dies, the remaining 3 are now at 107% and you cascade. I'd keep utilization below 60-70% so you can absorb a failure.

• Health checks: periodic probes, remove after N failures, re-add after recovery
• Retry on failure: transparent reroute for idempotent requests
• Graceful drain: stop new connections, finish in-flight, then shutdown
• Capacity headroom: keep utilization low enough to absorb node loss

Consistent hashing solves a specific problem: what happens when you add or remove servers from a pool and you need requests for the same key to go to the same server?

With regular modulo hashing — hash(key) % N servers — if N changes, almost every key remaps to a different server. If those servers have local state or caches, you just blew up everything. Cache hit rate drops to nearly zero during a scaling event.

Consistent hashing arranges servers on a ring. When you add or remove a node, only the keys that were assigned to that specific node get redistributed — roughly 1/N of the keys move instead of almost all of them. Way less disruption.

I'd use it for: distributed caches (like Memcached clusters), sharded databases where you want to add nodes without massive data migration, session stores where you need sticky routing to survive pool changes, and any stateful routing where cache locality matters.

The virtual nodes trick is important too — each physical server gets multiple positions on the ring. This prevents uneven distribution that happens with just N points on a ring, especially with small cluster sizes.

• Problem solved: minimize key remapping when cluster size changes
• How: hash ring where only ~1/N keys move on node add/remove
• Use cases: distributed caches, shard routing, session affinity
• Virtual nodes: multiple ring positions per server for even distribution

Oh there's a bunch. Load balancing is maybe 30% of why you'd run a reverse proxy. Let me walk through the other benefits.

TLS termination is huge — your backends don't need to handle SSL/TLS. The proxy handles encryption/decryption, certificate management, protocol negotiation. Your internal traffic can be plain HTTP (assuming a trusted network), which simplifies backend services and reduces their CPU load.

Caching at the proxy level — Nginx can cache responses and serve them without hitting the backend at all. For semi-static content, this is incredibly effective. One request builds the cache, thousands are served from it.

Request manipulation — adding headers, rewriting URLs, rate limiting, authentication checks before the request even reaches your application. You can enforce security policies at the edge. IP whitelisting, request size limits, blocking malicious patterns.

Compression — the proxy handles gzip/brotli so your application doesn't spend CPU on it. Plus connection pooling to the backend — clients might open many short-lived connections, but the proxy maintains a smaller pool of persistent connections to your servers. Way more efficient.

And abstraction — clients talk to one endpoint. Behind it you can swap, add, or restructure services without changing the public API. That's the real power for evolving your architecture.

• TLS termination: centralized certificate management, offload crypto from backends
• Edge caching: serve cached responses without backend round-trip
• Security: rate limiting, WAF rules, IP filtering, request validation
• Abstraction: hide internal topology, enable zero-downtime architecture changes
• Efficiency: compression, connection pooling, HTTP/2 multiplexing