Redundancy

Scalability is about absorbing load. Availability is about still being there when a component fails — and components always fail, so availability is bought with redundancy: more copies than you need, arranged so that losing one is uneventful.

A two-zone deployment with no single point of failure100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

• Availability: the share of time a system can serve requests, in "nines". A month is 43,200 minutes, so 99.9% allows about 43 minutes of downtime and 99.99% about 4.3 (terminology). That tenfold gap is the redundancy budget.
• Reliability: the neighbouring property — doing the job correctly during uptime, not being reachable.

Redundancy is duplicate capacity that takes over when a primary fails; its purpose is to leave no single point of failure, no component whose loss takes the system with it. Copies live at every layer, from a box's dual power supplies to app instances behind a load balancer and whole tiers duplicated across availability zones — the only layer that survives losing a building. A spare counts only if it is already running, which makes redundancy the raw material for fault tolerance.

Sizing the spare

A second copy is not automatically redundancy. Size from peak: 3,000 QPS, one box safely serving 500 QPS, so 3,000 ÷ 500 = 6 boxes carry the load. Split across two zones — three per zone — and each zone serves 1,500 QPS at 100% of its own capacity.

Now lose a zone. The three survivors inherit all 3,000 QPS against 1,500 QPS of capacity. Queues build, latency climbs, health checks fail on boxes that are merely saturated, the balancer pulls them, and the rest absorb more. The redundancy did not absorb a zone failure; it converted one into a total outage. Rate limiting and a circuit breaker help, but the real fix is arithmetic done in advance:

• 2N across two zones. Each zone alone carries the full 3,000 QPS: six boxes per zone, twelve in total, 50% utilised. Simple, at 200% of what you need.
• N+1 across three zones. Any zone lost leaves two carrying peak, so each is sized for 3,000 ÷ 2 = 1,500 QPS: three boxes each, nine in total, 67% utilised — 150% of need, and the overhead falls further as zones are added.

A cold standby started on failure is cheaper than both and least likely to work: it carries no traffic normally, so its first real execution is during the incident.

Redundancy that isn't

Two instances at 99% availability each give 1 − 0.01² = 99.99% for the pair — but only when their failures are independent, and often they are not.

• Shared blast radius. Both instances in one zone, behind one balancer, reading one configuration store. A pair that fails together is 99% available at twice the cost.
• The deploy. A bad binary reaches every replica within minutes. Redundancy covers one machine dying, not every machine being told the same wrong thing, so staged rollout and fast rollback buy more than a fourth replica.
• The component in series. Anything every request must pass — the balancer, DNS, an API gateway — multiplies into the result and caps the tier at its own availability. Duplicating the app tier under one balancer still removes the app tier's own downtime, but it cannot lift the tier past the balancer's number; the next nine has to come from a second balancer.
• The undetected failure. A replica that is up but answering with errors keeps taking traffic until a health check notices. Detection time is downtime, which is why monitoring is part of the availability number.

What a promoted follower costs

Replication is redundancy for data, and asynchronous replication means the copy is behind by a measurable amount. At 500 write QPS with 200 ms of p99 replication lag, 500 × 0.2 = 100 acknowledged writes exist only on the dead leader at the instant of promotion. Redundancy protects availability, not the durability of the last second of writes.

Clients retry what they lost, so the same request can be applied twice and the write path needs idempotency keys. Reads from a lagging follower break read-your-writes — a user updates their profile, the next read lands on the follower, and their change disappears — which makes it a consistency decision, not a footnote.

Synchronous replication closes the window by making every write wait for a follower to acknowledge. In the next zone that is a datacenter round trip, 0.5 ms, usually affordable; to another region it is roughly 200 ms on every write (India to US East), a different product. Choose per path — payments synchronous, activity events not — with the sizing from database scaling.

Redundant leaders bring a failure of their own: when the network partitions, each side can promote its own leader and take conflicting writes — split brain. The escape is a majority, so a quorum of an odd number of voters promotes and the minority side stops writing: consistency over availability during a partition, and, under PACELC's else, latency paid for it the rest of the time. See distributed consensus: it is why quorums are 3 or 5, never 2 or 4.

In an interview

What is being tested is whether you look at your own diagram and find the box with no twin. Walk the request path aloud, naming each single point of failure: "DNS, then an active/standby balancer pair, because one balancer caps the whole tier, then app instances across three zones, then a leader with a promotable follower in the second zone." Then size the survivors out loud — nine boxes across three zones, 67% utilised — and finish with what redundancy did not buy: promotion loses roughly the last 100 writes, so payments replicate synchronously.

The mistake that loses points is calling a component redundant without re-sizing the survivors. Two zones each at 100% of their own capacity are two ways to start a cascading overload, not a redundant pair. Close behind: a lone balancer in front of a duplicated fleet, and a "multi-AZ" design whose replicas all sit in one zone.

Check yourself

1. Peak is 6,000 QPS, one box safely serves 400 QPS, and losing a zone must change nothing users can see. How many boxes across two zones, and across four?

6,000 ÷ 400 = 15 boxes carry peak. Across two zones each must survive alone, so each needs all fifteen: 30 boxes, 50% utilised. Across four, three survivors carry peak, so each zone is sized for 6,000 ÷ 3 = 2,000 QPS — five boxes each, 20 in total, 75% utilised. The wider you spread, the smaller the spare. Eight per zone across two looks redundant and dies at roughly 190% of a survivor's capacity.

2. Two app servers, each 99.9% available, behind one load balancer that is also 99.9% available. What is the tier's availability, and where does the next nine come from?

The pair is down only when both are down: 1 − 0.001² = 99.9999%, so the app tier is not the limit. The balancer is in series, so the tier is 0.999 × 0.999999 ≈ 99.9% — about 43 minutes a month. Price the second server: one behind the balancer gives 0.999 × 0.999 ≈ 99.8%, about 86 minutes, so the second copy halved monthly downtime. A third buys nothing measurable; the next nine comes from duplicating the balancer. Series components set the ceiling; parallel copies help up to it, never past it.

3. One leader, one asynchronous follower, 500 write QPS, p99 lag 200 ms. The leader dies and the follower is promoted. What was lost, and what changes for payments?

500 × 0.2 = 100 acknowledged writes existed only on the dead leader — users told "saved" who were not. Clients retry, so the write path needs idempotency keys. For payments, replicate synchronously to a follower in the next zone (0.5 ms per write); another region costs roughly 200 ms per write, which is not.