Horizontal vs vertical scaling
Work out from QPS and availability numbers when to buy a bigger machine, when to add more of them, and what adding machines quietly costs.
Scalability is the ability of a system to handle a growing amount of work. There are only two ways to get it: make one machine bigger, or use more machines. Which one you pick decides everything downstream — whether you need a load balancer, whether a session can live in process memory, and whether a single reboot takes the product offline.
Vertical scaling (scaling up)
Vertical scaling increases the capacity of a single machine by adding resources: CPU, RAM, disk. It is like moving to a bigger house. You have more space, but it is still just one house.
It is the cheapest change you will ever make. Moving a server from 16 GB of RAM to 64 GB, or from 8 vCPUs to 32, is an instance-type change and a restart. The application code usually does not change at all — no load balancer, no shared session store, no distributed anything.
Three things stop it.
- A hard limit. The largest instance your cloud sells is the end of the road, and you reach it with no warning and no incremental fix.
- A single point of failure. One machine means every kernel patch, every deploy, and every disk failure is a full outage. A box that is up 99.9% of the time is down for 0.1% × 8,760 h ≈ 8.8 hours a year, and during those hours there is nowhere for traffic to go.
- Cost, and it is not linear. High-end hardware prices superlinearly — doubling the RAM often more than doubles the bill. Throughput does not double with core count either: lock contention, garbage-collection pauses, and single-threaded sections cap the gain well before the hardware does.
Horizontal scaling (scaling out)
Horizontal scaling increases capacity by adding more machines. It is like adding more houses to a neighbourhood: more capacity overall, and if one house has a problem the others still work.
There is no architectural ceiling in the application tier, the machines can be cheap commodity boxes, and losing one is survivable rather than fatal. The price is complexity: traffic must be distributed, data replicated or partitioned, and the application written to tolerate many identical copies of itself.
Sizing the decision
Assume 5 million daily active users making 20 requests each:
5,000,000 DAU × 20 req/day = 100,000,000 requests/day
100,000,000 ÷ 86,400 s ≈ 1,160 QPS average
peak at 3× average ≈ 3,500 QPS
Little's Law sizes the fleet: concurrency = arrival rate × latency. If a
request spends 40 ms inside the application, 3,500 QPS × 0.04 s = 140 requests
in flight. A 4-vCPU box that comfortably holds ~25 concurrent requests covers
that with 6 boxes; run 8 so one can fail or roll during a deploy without losing
capacity. That part is boring, and it is the easy half.
The hard half is the data tier. If each request runs 2 queries, peak database load is ~7,000 QPS, above the ~5,000 simple QPS a single commodity Postgres box sustains. No number of application servers fixes that. You cut reads with caching, then split the data with replication or sharding — see database scaling. Putting more stateless boxes in front of a saturated database moves the queue, it does not raise the ceiling.
What adding machines costs
Three costs are visible in that path.
Traffic has to be split, so a load balancer becomes a component of its own that you then have to make redundant.
State has to leave the process. A session held in local memory works with one box and breaks the moment a second box answers the next request, which is why horizontal scaling and stateless services arrive together.
Connections multiply, and this is the failure mode teams hit first. Each app box opens its own pool, so 20 boxes × 50 connections = 1,000 connections to a database that starts degrading in the low hundreds. Little's Law says the work does not need them: 3,500 QPS × 5 ms of database time ≈ 18 queries in flight, so 1,000 connections is roughly fifty times the concurrency the traffic actually requires. The result is that p99 latency gets worse after you add servers. The fix is a shared pooler and pool sizes derived from the arithmetic rather than from per-box defaults.
Read replicas relieve reads but not writes, and they lag. A user who posts a comment and immediately reloads can be served by a follower that has not applied the write yet, and sees their own update vanish. Reads that must be read-your-writes go to the primary — see consistency.
When to use which
Scale up while it lasts. Vertical scaling is the right answer for a small application with limited growth, or when a day of engineering costs more than the hardware. Scale out once either is true: you can no longer tolerate the outage window of one machine, or you are past half the largest instance available and have no headroom for a spike.
Production systems are usually both — a horizontally scaled fleet of vertically right-sized machines. Note that redundancy, not capacity, is often what forces the move: two boxes at 99.9% each are both down only 0.001² of the time, but only if their failures are independent. A shared database, one bad deploy, or a single rack correlates them and the arithmetic collapses back to one nine.
In an interview
You should be able to define both, and say that horizontal scaling is what large-scale distributed systems use because it gives high availability, fault tolerance, and no hard capacity ceiling. That is the baseline, and it is not what earns the point.
What the interviewer is testing is whether you can name the bottleneck before
choosing the remedy. Say which tier you are scaling and why: "the app tier is
stateless, so it scales out linearly — at 3,500 QPS peak that is about 8 boxes.
The constraint is the write path: one primary at roughly 5,000 simple QPS, so I
would add a read replica now and shard by user_id once writes pass ~2,000
QPS."
The mistake that loses points is answering "scale horizontally" as a reflex, with no bottleneck named — it tells the interviewer you have memorised the preference, not the reasoning. The second mistake is claiming horizontal scaling is unlimited. Coordination overhead, connection counts, and cross-shard fan-out all grow with the fleet, and past some size adding machines returns less than it costs.
Check yourself
1. A service takes 8 million requests a day and each request runs 3 queries against one Postgres box. Do you shard? Show the arithmetic.
No. 8,000,000 ÷ 86,400 ≈ 93 QPS average; at 3× peak that is ~280 QPS, and ×3 queries ≈ 840 database QPS — well under the ~5,000 QPS a single box handles. Sharding here buys nothing and costs you cross-shard queries and a rebalancing problem. Add a cache or a read replica when latency demands it.
2. You grow the fleet from 4 app servers to 20 and p99 latency gets worse. What is the most likely cause, and what do you change?
Connection amplification at the database. 20 boxes × a 50-connection pool = 1,000 connections, while the traffic only needs ~18 concurrent queries (3,500 QPS × 5 ms). The database spends its time on connection overhead instead of queries. Put a shared pooler in front and size pools from Little's Law, not per-box defaults.
3. Your single server is available 99.9% of the time. How much downtime is that per year, and what does a second server behind a load balancer actually buy you?
0.1% × 8,760 h ≈ 8.8 hours a year, about 43 minutes a month. A second server removes the machine as a single point of failure, but only for independent failures and only if the application is stateless. If both servers share one primary database, or take the same bad deploy, the failures are correlated and your real availability is that of the shared dependency.