Key terminology for distributed systems
Define latency, throughput, CAP, PACELC, ACID and replication precisely enough to defend a design choice out loud, and stop conflating the two Cs.
Latency is a duration; throughput is a rate. A distributed system can improve one while making the other worse, and design arguments are routinely lost by candidates who use the two words interchangeably. Every term below is defined tightly enough to decide with.
Latency and throughput move independently
Latency is how long one request takes, end to end. Quote a percentile, never a mean: p99 is what users feel, and the mean hides the tail that files the support ticket.
| Operation | Time |
|---|---|
| Main memory reference | 100 ns |
| SSD random read | 100 µs |
| Round trip inside one datacenter | 0.5 ms |
| Disk seek (spinning) | 10 ms |
| Round trip India to US East | ~200 ms |
A cross-region round trip costs roughly 400 times a same-datacenter one. That ratio decides more designs than any other number here.
Throughput is requests served per unit of time. The conversion you will do in almost every interview:
1,000,000 req/day / 86,400 s ~= 12 QPS average
1,000,000,000 req/day / 86,400 s ~= 12,000 QPS average
peak is typically 2-5x average
Little's Law ties them together: concurrency = arrival rate x latency. At 500 QPS with 200 ms latency, 500 x 0.2 = 100 requests are in flight, so a pool of 50 connections queues — which is why a slow dependency exhausts a pool long before it exhausts CPU. And horizontal scaling raises throughput without lowering one request's latency; candidates routinely claim it does both.
CAP: what you do during a partition
The popular phrasing — "pick two of the three" — is a misstatement. The properties are real: Consistency, every read observes the most recent write; Availability, every request gets a non-error response with no guarantee it is current; Partition tolerance, the system keeps operating when messages between nodes are dropped or delayed.
But P is not a property you choose. Links fail whether or not your design approves, so anything spread across more than one machine must tolerate partitions. There is no useful CA system; that label describes a single node.
CAP's real content is a decision that comes due only during a partition: with the link down, does a node refuse the request rather than serve stale data, or answer anyway?
Both branches are defensible, and the choice belongs to the data, not the architecture. A payment ledger takes CP: a stale balance is worse than an error. A follower count takes AP: a four-second-old number harms nobody, an error page harms everyone.
PACELC completes it, and its second half applies almost all the time: if Partition, choose Availability or Consistency; Else, choose Latency or Consistency. Partitions are rare, so the else-branch is the normal condition, and it still costs you. Making a read in Mumbai observe a write committed in Virginia means paying that ~200 ms round trip, against 0.5 ms served locally and stale. Consistency is billed in latency every day, not only on the bad day.
ACID, BASE, and the two Cs that are unrelated
ACID is a relational database's transaction guarantee. Atomicity: every operation in a transaction succeeds or none does. Consistency: the transaction moves the database from one valid state to another, preserving its invariants. Isolation: concurrent transactions behave as if run one after another. Durability: a committed write survives a crash.
BASE is the posture most NoSQL stores take instead: Basically Available, Soft state (data changes as replicas converge, with no new input), Eventually consistent (reads may be stale, but replicas converge once writes stop).
Say this out loud, because it is conflated constantly: the C in ACID and the C in CAP are unrelated properties that share a letter. ACID-C is about invariants — a transfer never leaves money in neither account. CAP-C is about recency — every read sees the latest write. A database can satisfy ACID-C on every node and still hand you a stale replica read. See consistency models for the ladder between strong and eventual.
Replication: leader and follower
Replication copies data to more than one machine for availability and read capacity. Under leader/follower (also primary/replica), one node takes writes and streams changes to followers that serve reads. Under multi-leader, several nodes take writes and replicate to each other, buying write availability and handing you conflicts to resolve.
Serving reads from followers is the standard way to scale a read-heavy workload, and it carries one consequence you must name before the interviewer does. Followers lag — milliseconds normally, seconds under load. A user updates their profile, their next read lands on a follower that has not caught up, and the change appears to vanish. That is read-your-writes breaking, and it arrives as a bug report saying the save button does nothing. Route that user's reads to the leader for a few seconds after a write, or pin the session to one replica.
Sharding (partitioning) splits one dataset across machines by key. It answers a dataset or write rate that will not fit on one box; it is not the first answer to a read problem. A commodity Postgres box handles roughly 5,000 simple QPS, so at 1,000 QPS peak the honest answer is "not yet, and here is the number that would change my mind." See scaling the data layer.
The components you will name in every design
- Load balancing: spread traffic so no instance saturates and a dead one stops receiving requests.
- Caching: hot data in fast storage. At 1,000 QPS peak with a 95% hit rate the database sees 50 QPS, not 1,000.
- Proxy: a forward proxy acts for clients (egress control, privacy); a reverse proxy acts for servers (TLS termination, caching, load balancing).
- API gateway: one entry point in front of many services, owning routing, auth and rate limiting.
- Message queue: asynchronous handoff, so a slow consumer no longer blocks the caller.
- Microservices vs monolith: services split by business capability versus one deployable unit. The monolith is the right default until team size or scaling asymmetry pays for the added network calls.
Idempotency means applying a request twice gives the same result as once. A durable queue almost always delivers at-least-once, at-least-once means duplicates arrive, and duplicates mean every consumer needs an idempotency key or you will double-charge somebody.
Availability is the fraction of time the system serves requests. Convert the nines rather than quoting them: a month is 30 x 24 x 60 = 43,200 minutes, so 99.9% allows about 43 minutes of downtime and 99.99% about 4.3 — a tenfold cut in error budget, usually bought with redundancy in a second zone.
In an interview
The interviewer is testing whether these words are load-bearing for you or decorative. Terminology scores nothing on its own; it scores when a decision hangs off it. Use each term to justify, never to garnish. Not "we need consistency here" but "this is a balance read, so during a partition I take the CP branch and return an error rather than a stale number — and outside a partition, PACELC still charges 200 ms per cross-region read, so the ledger stays single-region."
Two mistakes reliably lose points. The first is "CAP means you pick two of three"; partitions are not optional, so the only choice is what you do during one. The second is treating ACID-C and CAP-C as one property — a candidate who says "we use Postgres, so we get CAP consistency across replicas" has announced they have never run replicas in production.
Know why each term exists and which decision it informs. A definition recited with no trade-off attached reads as memorisation, and the round is built to see through it.
Check yourself
A shopping cart takes 20 million requests a day. During a partition between zones, should add-to-cart succeed against a replica that cannot reach the leader?
Yes — the AP branch.
20,000,000 / 86,400 ~= 230 QPSaverage, about 1,000 QPS at a 4x peak, well inside one primary. A cart is recoverable state, and merging at partition heal (union the items) beats refusing add-to-cart during an outage. Checkout is the opposite: a duplicated payment cannot be repaired by a merge, so it takes CP.
Your replicas lag 300 ms. A user renames a photo album and is immediately shown the list from a replica. What do they see, and what do you change?
The old name. Read-your-writes has broken, and it gets filed as a bug where saving does nothing. Route that user's reads to the leader for a few seconds after their write, or pin the session to one replica.
A service calls a dependency whose p99 is 400 ms, at 250 QPS. How many requests are in flight, and how big is the pool?
250 x 0.4 = 100concurrent requests, by Little's Law. A pool of 100 is break-even with zero headroom, so size above it and set a timeout well under 400 ms. If that dependency degrades to 2 s, the same 250 QPS needs 500 slots, the pool saturates, and a service that is not itself broken goes down with it.