Load balancing

A load balancer is the single address clients aim at, and the component that decides which of N interchangeable servers answers each request. It is what makes horizontal scaling usable: adding a server only helps if something routes work to it. Two decisions define the design — what the balancer inspects before choosing a backend, and how fast it notices a backend has died.

The request path

A load balancer behind a DNS virtual IP, terminating TLS and ejecting an unhealthy server100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

The balancer is almost always software now: a managed cloud service (GCP's HTTP(S) load balancer, AWS ALB at layer 7, NLB at layer 4) or self-run (Envoy, HAProxy, NGINX). Hardware appliances survive mainly at the datacentre edge.

Layer 4 vs layer 7

A layer 4 balancer sees source IP and destination port, picks a backend when the TCP connection opens, and forwards packets for the life of that connection. It is fast because it never parses a byte of payload — and for the same reason it cannot route on a URL, rewrite a header, or retry a failed request, having no idea where one request ends.

A layer 7 balancer terminates the connection, parses HTTP, and chooses per request. That buys content-based routing (/images to one pool, /api to another), header manipulation, retries for idempotent calls, and multiplexing thousands of client connections onto a few keep-alive connections per backend. The cost is CPU and a few hundred microseconds, which against a 0.5 ms in-datacentre round trip is rarely the constraint. Default to layer 7 for HTTP; layer 4 for other protocols or raw packets-per-second.

Choosing an algorithm

Algorithm	Routes on	Choose it when
Round robin	Next in sequence	Requests cost about the same; one call in a hundred at 40× the median breaks it
Least connections	Fewest in-flight	Cost varies — with a 5 ms read and a 2 s export, round robin piles up on whichever box drew the exports
Weighted round robin / least connections	The above, scaled per server	Hardware is heterogeneous, usually mid-migration
IP hash	hash(client_ip) mod N	You need stickiness and cannot fix the state problem yet
URL hash	hash(path) mod N	Backends hold a local cache and one URL should stay on one hot box
Resource-based	Lowest reported CPU	Least connections proved insufficient; it oscillates, herding traffic onto whichever box last reported low CPU

Sizing the pool

Assume 1M DAU at 20 requests/day: 20M ÷ 86,400 s ≈ 230 QPS average, and at a 4× peak factor roughly 1,000 QPS peak.

Now size one server. Say 16 worker threads and a 50 ms mean service time. Little's Law run backwards — concurrency = QPS × latency, so 16 = QPS × 0.05 s — gives 320 QPS before requests queue. Derate to 250 QPS for GC pauses and the tail. 1,000 ÷ 250 = 4 boxes at full utilisation, so run 6: utilisation is 67%, and losing one still leaves 5 × 250 = 1,250 QPS against a 1,000 QPS peak.

Provisioning the minimum 4 is the failure mode worth naming. Lose one and 750 QPS of capacity faces 1,000 of demand. Queues fill, p99 goes vertical, health checks start timing out on the survivors, and the balancer ejects them too. One node failure becomes a full outage because the pool had no slack.

Health checks and how fast you eject

The balancer probes each backend and removes the ones that fail. A probe can be a ping, a TCP connect, an HTTP request checked for 200, or a custom endpoint verifying application-specific health.

The timing is what candidates skip. A common default — probe every 5 s, eject after 3 consecutive failures — leaves a dead box taking traffic for up to 15 s. At 1,000 QPS over 6 boxes it owns ~167 QPS, so roughly 2,500 requests fail before ejection. A 1 s interval with 2 failures cuts that to ~2 s and ~330. Nobody goes tighter still, because aggressive probes eject healthy boxes that paused for GC, the smaller pool is then more loaded, and more boxes fail. Envoy guards this with a panic threshold: once more than 50% of the pool is unhealthy it ignores health status entirely, on the theory that a degraded pool beats an empty one.

Probe depth carries the matching trade. A check answering 200 from a static handler stays green while the box's database pool is exhausted, so traffic keeps flowing to a server that fails every real request. A deep check touching the database fails on every box at once during a single blip, and the pool ejects itself. The compromise: shallow check for the balancer, deep check wired to monitoring.

Session persistence, and the failure it causes

Some applications need every request from a client on the same server, because the session lives in that server's memory. IP hash and cookie-based sticky sessions (the balancer sets a cookie naming the backend and honours it later) both do that, and share one failure mode worth stating precisely. A balancer computing hash(client_ip) mod N remaps clients whenever N changes. Go from 6 boxes to 5 and it is not 1/6 of sessions that move: almost every remainder changes, so nearly all users are logged out at once and the fleet takes a cold-cache stampede on the way back. Consistent hashing bounds that to about 1/N, and is the fix if you must hash.

The better fix is to stop needing stickiness: keep app servers stateless and hold session data in a shared store such as Redis. Any box then serves any request, any algorithm becomes legal, and a dead box costs one in-flight request instead of a session — for one extra hop of 0.5 ms against a 50 ms service time, about 1% of the budget.

TLS termination

Decrypting HTTPS at the balancer centralises certificate management and is a precondition for layer 7 routing: you cannot route on a URL you cannot read. The cost sits in handshakes, not bulk encryption. A full handshake costs roughly 1–2 ms of CPU on the terminating side, so 1,000 QPS with no connection reuse means 1,000 handshakes/s — two cores doing nothing but key exchange. Keep-alive amortising ~100 requests per connection drops that to 10 handshakes/s. Connection reuse, not cipher choice, is the number to check. The backend leg can then run plain HTTP inside a trusted network, though most organisations now re-encrypt it.

The balancer is a single point of failure

Every request crosses one component, so a lone balancer is the thing that takes the site down. Run at least two in an active-active or active-passive pair sharing a virtual IP, with DNS or anycast in front so a whole zone can drop out. DNS failover alone is slow: resolvers cache for the TTL, so a 300 s TTL leaves up to five minutes of traffic aimed at a dead address. DNS handles zone and geographic distribution; VIP failover handles instance-level failure — the pattern generalises as redundancy.

In an interview

The interviewer is testing whether a load balancer is a box you draw or a component you can reason about. The signal is in the layer, the algorithm, the check policy, and the capacity.

Say the layer and why: "layer 7, so we can split /api and /static onto separate pools and retry idempotent GETs." Name the algorithm with the workload claim behind it: "least connections, because our export endpoint is seconds and our read endpoint is milliseconds." Size the pool out loud with Little's Law rather than asserting a server count, state the check interval with its eject-window arithmetic, and say the balancer is redundant before you are asked. Know the managed offerings by name for the company you are interviewing with — for Google, GCP's cloud load balancers.

Three mistakes lose points. Answering "sticky sessions": it works, and it says you would rather configure the balancer than fix the architecture. Drawing one load balancer: that is a single point of failure. Naming an algorithm with no workload claim: "round robin" is not a decision until you have said requests cost about the same.

Check yourself

Peak is 1,000 QPS. Each box runs 16 worker threads at a 50 ms mean service time. How many boxes, and why not the minimum?

16 ÷ 0.05 s = 320 QPS per box; derate to ~250. 1,000 ÷ 250 = 4 at full utilisation, so run 6. Losing one of 6 leaves 1,250 QPS against a 1,000 QPS peak; losing one of 4 leaves 750 against 1,000, so queues fill and the balancer ejects the survivors too.

Checks run every 5 s and eject after 3 failures. A box dies at peak. How many requests fail, and what does tightening cost?

Up to 15 s of traffic keeps hitting it; at ~167 QPS per box, roughly 2,500 failures. A 1 s interval with 2 failures cuts that to ~2 s and ~330, at the risk of ejecting healthy boxes that paused for GC — which shrinks the pool and can cascade.

Six boxes use hash(client_ip) mod N for stickiness and one fails. What fraction of users lose their session, and what should you have built?

Close to all of them, not 1/6: N goes 6 to 5, so nearly every remainder changes. Consistent hashing would hold it near 1/6. Better still, do not hash — put session state in a shared store and let any box serve any request, for one 0.5 ms hop.