Load balancers and how they pick a server
Size a server pool with Little's Law, set health checks that catch a sick server without ejecting the fleet, and know what sticky sessions cost.
A load balancer is a device or software application that distributes network or application traffic across a cluster of servers. It sits between the client and the pool, acting as a reverse proxy: it terminates the client's connection, picks a server, opens its own connection to that server, and passes the response back. That indirection is what makes horizontal scaling possible — clients hold one address, and we add or drain servers behind it without telling anyone.
The request path
- The client's request arrives at the load balancer, not at a server.
- The load balancer picks a server from the pool it believes is healthy.
- It forwards the request, takes the server's response, and returns it to the client.
Step 2 holds the two ideas worth learning: which server, and believes is healthy.
Because the load balancer terminates the connection, the backend sees its IP, not the client's — the original travels in an X-Forwarded-For header, so anything that rate-limits by client IP reads that header instead of the socket. The same property lets it terminate TLS once for the whole fleet, as proxy servers covers.
Choosing a server
- Round robin. Each server in turn. Correct only when requests cost the same: if one endpoint takes 2 s and the rest take 20 ms, round robin hands the slow request to a server already carrying three of them.
- Least connections. The server with the fewest in-flight requests wins — the sane default when costs vary, because in-flight count tracks real load.
- IP hash. The client's IP decides the server, so a client keeps landing on the same one. Useful for session persistence, and fragile: a corporate NAT gateway is one IP, so its whole office hashes onto one server.
- Weighted variants. Each server carries a weight, so a box with twice the cores takes twice the traffic. This keeps a mixed-generation fleet balanced.
- Resource-based (adaptive). Routes on reported CPU or memory — closest to real load, with a feedback loop that oscillates when the signal lags.
Sizing the pool
Take a service with 2 million daily active users making 20 requests a day:
2,000,000 DAU x 20 req/day = 40,000,000 req/day
40,000,000 / 86,400 s ~= 460 QPS average
460 QPS x 4 (peak factor) ~= 1,850 QPS peak
Now size one server with Little's Law — concurrency = arrival rate x latency. Assume a server holds 100 requests in flight before queueing, and a request takes 200 ms:
100 in flight = arrival x 0.2 s -> 500 QPS per server
1,850 QPS peak / 500 -> 3.7, so 4 servers
Four servers carry peak exactly, and that is the trap. Lose one to a bad deploy and the remaining three absorb 1,500 QPS against 1,850 arriving: 350 requests a second pile up, in-flight count climbs past 100, p99 crosses the client timeout, and the survivors start failing their own health checks. One dead server becomes zero live ones in under a minute. Provision N+1 — five — so a failure still leaves 2,000 QPS of capacity against 1,850 of demand. That is the real argument for horizontal scaling.
Health checks, and the two ways they go wrong
A load balancer only knows a server is sick because it probes it. The naive probe opens a TCP connection every 30 s and ejects after three failures — and it misses the most common failure. A server whose database connection pool is exhausted still accepts TCP connections, so it stays in rotation, takes a fifth of the traffic and fails all of it. A /healthz endpoint that exercises the dependencies returns a 500 instead.
The detection window is arithmetic. A 2 s interval with a 3-failure threshold means up to 6 s of errors; at 1,850 QPS across five servers, that one holds about 370 QPS, so roughly 2,200 requests fail before ejection. Probing every second with a 2-failure threshold cuts it to about 740.
The opposite failure is worse. If the probe fails whenever the database is slow, a two-second stall fails every server's check at once, and a load balancer obeying its own rules ejects the entire pool — a degraded system becomes a total outage. Production load balancers guard this with a panic threshold: when less than about half the pool is healthy, ignore health status and spread traffic across all of it, on the grounds that a degraded server beats no server.
Session persistence, and why we avoid needing it
A load balancer can pin a client to one server — via IP hash, or a cookie it sets — so in-memory session state stays reachable. It works, and it costs: with five servers, losing one logs out 20% of active users, and pinning overrides least-connections balancing.
The alternative is to keep no session state on the server and put it in a shared store. A Redis lookup adds about 0.5 ms — one datacenter round trip — against a 200 ms budget, or 0.25%. Any server can then serve any request, which is what makes draining and autoscaling boring. See stateless services and caching.
The load balancer is now the single point of failure
Five servers behind a load balancer means one server failure no longer takes the site down — and every request now passes through one box. Availability moved; it did not disappear. The answers are a pair of load balancers sharing a floating virtual IP, several load balancer addresses under one name so DNS hands out alternatives, or anycast. Managed services do this internally; redundancy covers the pattern.
Types
Hardware appliances buy throughput at high price and months of lead time. Software load balancers (HAProxy, Nginx, Envoy, Traefik) are cheap and flexible and leave you the redundancy problem above. Cloud-managed services arrive already redundant, at the cost of provider-specific health check and timeout behaviour.
In an interview
Every candidate draws the box, so drawing it scores nothing. The interviewer is testing whether you know what it decides, and what happens when it decides wrong.
Give the algorithm with a reason: "Least connections, because request costs range from 20 ms reads to 2 s exports, and round robin would queue slow behind slow." Give the health check as a number: "Probe /healthz every 2 s, eject after 3 failures, so a sick server serves errors for at most 6 s." Give the redundancy: "The load balancer is a pair with a floating IP, otherwise I moved the single point of failure rather than removing it."
The mistake that loses points is claiming the load balancer gives you high availability while provisioning N servers for exactly N servers' worth of peak. If losing one puts the rest over capacity, you have not added availability — you have built a cascading failure that arrives one node at a time. The follow-up is always "what happens when one dies?", and the answer has to be a number.
Check yourself
A service peaks at 3,000 QPS. Each server holds 150 requests in flight and averages 300 ms per request. How many servers do you run?
Little's Law gives per-server capacity: 150 in flight / 0.3 s = 500 QPS. 3,000 / 500 = 6 servers to carry peak, so run 7. Losing one then leaves 3,000 QPS against 3,000 of demand — exactly at the edge — so if the peak factor is uncertain, 8 is the defensible answer, and you say why.
Your health check queries the primary database. The database has a 3-second stall. What does the load balancer do, and what should it have done?
Every server fails the probe in the same window, so the load balancer ejects the whole pool and returns 503 to everyone: a 3-second stall becomes a full outage, plus a stampede when servers rejoin. The probe should test only what that server controls, with dependency health reported separately, and the load balancer needs a panic threshold that ignores health status when most of the pool looks unhealthy.
You are told to keep sessions in server memory and use IP hash for persistence. Argue for or against, with the consequence named.
Against, in most designs. IP hash pins every client behind one NAT gateway onto a single server, so balance degrades exactly when a large customer arrives, and losing a server destroys the sessions it held — with 5 servers, 20% of users are logged out. A shared store costs about 0.5 ms per request, 0.25% of a 200 ms budget, and buys draining, rolling deploys and real least-connections balancing. Keep stickiness only when the state is too large to externalise, such as a long-lived WebSocket.