Stateful vs stateless services
Move session state off the app servers, price the extra hop against a 50 ms request budget, and name the services that stay stateful whatever you do.
A stateless service keeps nothing about a client between requests, so any server in the fleet can answer any request and a dead server costs one in-flight request rather than a user's session. A stateful service remembers, which means one specific server has to answer, which means the load balancer, the deploy process and every failure now have to respect that binding. The distinction decides whether adding a box actually adds capacity, so it sits directly under horizontal scaling.
The two shapes
A stateful service remembers information about past interactions — "state" — on the server that handled them. A shopping cart in process memory, a game server tracking each player's position, and a database are all stateful. Requests from that client must return to the same server, which is what session persistence means.
A stateless service treats each request independently: everything needed arrives in the request itself — URL, headers, body, token — so the server looks nothing up about who asked last time. A web server returning static HTML, an API computing an answer from its inputs, and a read endpoint that queries a database but holds no per-client memory are all stateless.
Stateless does not mean the system has no state. Carts, sessions and balances still exist. It means the state does not live in the app tier: it moved to a store built to replicate and fail over, and the app servers became interchangeable.
| Stateful app tier | Stateless app tier | |
|---|---|---|
| Routing | client pinned to one server | any server, any request |
| Adding a server | starts empty; state must be shared or replicated | takes traffic immediately |
| Losing a server | every session on it is gone | one in-flight request, retried elsewhere |
| Deploys | a restart drops that box's sessions | rolling restart is invisible to users |
| Debugging | the outcome depends on which box you hit | the request replays identically anywhere |
The same fleet, in numbers
Assume 1M DAU making 20 requests/day: 20M ÷ 86,400 s ≈ 230 QPS average, and roughly 1,000 QPS peak at a 4× peak factor, served by 6 app servers — about 167 QPS each.
How many sessions are live at once? Little's Law answers it: 1M sessions/day ÷ 86,400 s ≈ 12 new sessions/s, and if a session lasts about 10 minutes, 12 × 600 s ≈ 7,000 concurrent sessions on average, call it 28,000 at peak. At 2 KB each that is ~56 MB of session data for the entire product. One Redis instance holds it with room to spare; there is no sharding argument to have.
Now break one box at peak. With sessions in process memory, 28,000 ÷ 6 ≈ 4,700 users lose their carts and are logged out, on top of the ~167 QPS of errors while the balancer ejects the box. With the sessions in a shared store, the same failure costs the ~167 requests that were in flight, all retryable, and nobody notices.
Deploys make the point harder, because they are scheduled rather than unlucky. A rolling restart touches all 6 boxes, so a team shipping 5 times a day destroys every session in the fleet 5 times a day.
What moving the state out actually costs
One extra round trip inside the datacentre is 0.5 ms. Against a 50 ms request budget that is 1%. Run it through Little's Law at the box level and the extra hop adds 167 QPS × 0.0005 s ≈ 0.08 concurrent requests — it does not register.
The number that does bite is distance. A session store in another region turns that 0.5 ms into a ~200 ms India-to-US-East round trip and a 50 ms request into a 250 ms one. Keep the store in the same region as the servers reading it.
Load on the store is equally undramatic: 1,000 QPS of lookups at peak against a Redis instance serving tens of thousands of operations per second. Postgres also works and survives a restart, but a commodity box handles roughly 5,000 simple QPS, so 1,000 QPS of session reads spends about 20% of that budget on the least interesting query in the system. Use a cache for the hot path, a database only when the session must outlive it.
Four places to keep the state
Sticky sessions. The balancer pins a client to a box. Cheapest and weakest: hash(client_ip) mod N remaps almost every client when N goes from 6 to 5, so one dead box logs out nearly the whole fleet — the arithmetic is in load balancing. A migration bridge, not a design.
Session replication. Copy each session write to every other box. It removes the affinity requirement and pays in write amplification: 200 session writes/s to 5 peers is 1,000 extra messages/s, and since every box replicates to every other, fleet-wide cost grows with N × (N-1). Workable at 3 boxes, unusable at 30.
A shared cache (Redis, Memcached). The default answer. One hop, one place to expire sessions, and box death stops being interesting. Login now depends on it, so it needs its own redundancy — a single unreplicated Redis has only moved the single point of failure.
A database. The same shape, durable, slower, and capped near 5,000 simple QPS per box. Right when sessions carry money or must survive a full cache flush.
The alternative that removes the lookup
Sign the session into the token itself — a JWT in a cookie — and there is no store to read. The server verifies a signature in tens of microseconds and answers. What you save in latency you pay in revocation: nothing the server does retracts a token already in the wild. Ban a user or strip a permission and a 15-minute token keeps working for up to 15 minutes. The usual repairs — a short TTL plus a refresh call, or a denylist checked per request — put back most of the lookup you were avoiding. Take tokens when 15-minute staleness is acceptable, and a session store when logout has to be immediate; the trade-off in full is under authentication and authorization.
Services that cannot be stateless
Some state refuses to move. A WebSocket connection is state by construction: pinned to one process, and when that process dies the client reconnects elsewhere and rebuilds its subscriptions from a shared store. Databases, game servers and stream processors holding time windows sit in the same place.
The goal was never to eliminate state, but to concentrate it in a few components designed to replicate and fail over, and keep everything in front of them interchangeable. That interchangeability is what lets microservices scale and deploy independently.
In an interview
The interviewer is testing whether "we will add more servers" is a real answer or a wish. Stateless app servers are what make it real, so advocate for them and say why in terms of load balancing, failure and deploys rather than as a slogan.
When the design needs state, say where it goes and what the hop costs: "sessions in Redis, same region, 0.5 ms against a 50 ms budget, so any box serves any request and losing one costs a single request." Size it out loud — 28,000 concurrent sessions at 2 KB is 56 MB, which settles the "do we shard the session store" question before it is asked.
Three mistakes lose points:
- Answering "sticky sessions". It works, and it says you would rather configure the balancer than fix the architecture. Mention it only as a bridge.
- Claiming stateless means the system holds no state. The interviewer will ask where the cart lives, and there is no good recovery.
- Reaching for signed tokens without naming revocation. Say the staleness window and how you would bound it.
Check yourself
Six app servers hold ~28,000 concurrent sessions in process memory at peak. One box dies. How many users are affected, and what changes if the sessions live in Redis?
About
28,000 ÷ 6 ≈ 4,700users are logged out and lose their carts, plus roughly 167 QPS of errors until the balancer ejects the box. With a shared store, only the ~167 in-flight requests fail and they retry on another box. The same reasoning covers deploys, which hit all six boxes on purpose.
Someone objects that a Redis round trip per request is too expensive. Settle it with arithmetic.
0.5 ms in-datacentre against a 50 ms request budget is 1%, and by Little's Law it adds
167 QPS × 0.0005 s ≈ 0.08concurrent requests per box. The objection only becomes right if the store sits in another region — a ~200 ms round trip would turn a 50 ms request into 250 ms.
An admin console must be able to revoke a user's access immediately. Signed token or shared session store?
Session store. A signed token cannot be recalled, so a 15-minute TTL leaves a revoked user active for up to 15 minutes. Keeping tokens and adding a denylist checked on every request restores immediacy but reinstates the per-request lookup, which was the reason to choose tokens in the first place.