Servers and what limits them

A server is a computer, or a software process on one, that provides services or resources to clients. It is the responder in client-server architecture: it listens on a port, accepts requests from clients, processes them, and sends responses back. The question worth answering is not what a server is, but how much traffic one absorbs before it stops responding and what breaks first when it does.

The loop every server runs

A server process repeats four steps forever.

1. Accept. The kernel completes the TCP handshake and hands the process a socket. Connections arriving faster than the process accepts them wait in the listen backlog; when it fills, the kernel drops new attempts and the client sees a timeout, not an error page.
2. Parse. Read bytes off the socket and turn them into a request: method, path, headers, body.
3. Execute. Run the handler — validate input, query a database, call another service, render a response.
4. Respond. Write bytes back, then close the connection or keep it alive for the next one.

Nearly all the latency lives in step 3; nearly all the capacity limits live in steps 1 and 3.

The three tiers

The classic split gives those steps their own machines.

• Web servers (Nginx, Apache, IIS) terminate TLS, serve static files, and forward dynamic requests upstream, tuned for many cheap connections.
• Application servers (Tomcat, JBoss, a Node or Go process) hold the business logic — the code you wrote, and where request latency is usually spent.
• Database servers (PostgreSQL, MySQL, MongoDB, Cassandra) own durable state and answer queries; which kind you pick is covered in SQL vs NoSQL.

Other roles run the same loop over a different protocol: mail servers (Postfix, Exchange), file servers (NFS, SMB/CIFS), DNS servers, proxies, and game servers holding session state.

The tiers a request crosses inside a server-side system100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

When your browser asks Google for a page: a web server takes the request, an application server processes it, a database server returns the data, and the response walks back. The hop-by-hop version is in the request walkthrough.

Sizing one box

Assume 1 million daily active users at 20 requests each:

1,000,000 × 20 ÷ 86,400 s ≈ 230 QPS average

Peak is typically 2–5× average, so plan for ~1,000 QPS.

Little's Law turns that into a thread count: concurrency = arrival rate × latency. Say a handler spends 50 ms — two datacenter round trips at 0.5 ms each, a Redis read, an indexed Postgres query, some CPU:

1,000 QPS × 0.05 s = 50 requests in flight

Fifty concurrent requests fit comfortably in a server sized at 200 threads. Read the equation backwards for the ceiling: 200 threads ÷ 0.05 s = 4,000 QPS before the pool is the bottleneck. That is the same order as the ~5,000 simple QPS a single commodity Postgres box handles, which is why one app box and one database box carry a real product further than most people expect.

The 50 ms must be a percentile, not a mean: p99 is what users feel and what sizes the pool.

The failure mode: a slow dependency eats the pool

A schema change drops an index and database p99 goes from 50 ms to 2 s. Nothing crashed. Apply Little's Law again:

1,000 QPS × 2 s = 2,000 requests in flight

The pool holds 200, so 1,800 queue and the backlog grows by roughly 800 per second. Within seconds every arriving request waits behind requests that will time out anyway. If the health check comes from that same pool it times out too, so the load balancer marks healthy servers dead and shifts their traffic onto the survivors, which fail identically. One slow query became an outage.

The tempting fix is to raise the pool to 2,000 threads. Reject it: 2,000 stacks at ~1 MB each is 2 GB before any heap, context switching climbs, and the database now sees 2,000 concurrent queries instead of 200, which makes it slower still. What works is a request timeout well below the client's patience, a bounded queue that sheds load with a 503, health checks on their own port, and a circuit breaker that fails fast while the dependency is sick. A cache in front of the query shortens step 3, shrinking the concurrency term itself.

Thread per request, or an event loop

Thread-per-request is simple: blocking code, stack traces that read top to bottom, one request per thread. At ~1 MB of stack each, a few hundred to a couple of thousand threads is the practical ceiling.

An event loop (Node.js, Nginx) multiplexes many connections over a few OS threads. It wins when connections are numerous and mostly idle — 20,000 open WebSocket connections need 20,000 threads in the first model, a few kilobytes of state each in the second. It loses on CPU-bound work, because one slow handler stalls every other connection on that loop. See concurrency.

Scaling, and why state is the obstacle

Vertical scaling means a bigger box: no code changes, one ceiling, one failure domain. Horizontal scaling means more boxes behind a load balancer — the only route past a single machine's limit, but it requires that the server keep no request-specific state in local memory. If a session lives in one process's heap, the next request routed elsewhere will not find it. Move that state to a shared cache or database and any box serves any request. See stateful vs stateless and horizontal vs vertical scaling. Statelessness also buys rolling deploys and cheap host failure: losing a box costs only its in-flight requests, which is usually why a third server exists.

Servers run server-oriented operating systems (Linux, Windows Server) and face the internet, so every byte read in step 2 is untrusted until validated (common threats).

In an interview

The interviewer is testing whether you can turn "add a server" into a number and a named failure mode. Say what one box absorbs, then what breaks first — rarely CPU, usually the thread or connection pool, the listen backlog, or the database behind it.

Phrasing that lands: "At 1,000 QPS peak with a 50 ms p99, that's 50 concurrent requests by Little's Law, so one app box at 200 threads has 4× headroom. I'd still run three across availability zones — for rolling deploys and host failure, not throughput."

The mistake that loses points is reciting the taxonomy — web, application, database, mail, proxy — as if naming the categories were the answer, then saying "we'll scale horizontally" without saying what makes the server stateless or where the ceiling is. The runner-up is quoting average latency; the interviewer will ask about the tail, and the mean hides it.

Check yourself

1. Your pool holds 300 threads, p99 per request is 40 ms, and a launch pushes peak traffic to 9,000 QPS. Add threads, add boxes, or both — and how many?

Little's Law: 9,000 × 0.04 s = 360 concurrent requests, against a one-box ceiling of 300 ÷ 0.04 = 7,500 QPS. Adding threads is the wrong lever — the CPU and the database behind it are sized for 300 in flight. Add boxes: two carry 180 concurrent each, and three means losing one at peak still leaves 15,000 QPS of ceiling. Then check the database, because 9,000 QPS against a single commodity Postgres box (~5,000 simple QPS) is now the real bottleneck.

2. A colleague wants to move the app servers to a region near users, 200 ms from the database, to cut perceived latency. The page issues 30 sequential queries, each 0.5 ms of network plus 1 ms of execution. What happens?

Data access costs 30 × 1.5 ms = 45 ms today. After the move each query pays a cross-region round trip: 30 × 201.5 ms ≈ 6 s. The change trades one 200 ms round trip for thirty. Keep the app server beside the data it queries, and put a CDN or edge cache in front of the user instead.

3. A chat feature adds 20,000 mostly idle WebSocket connections to a thread-per-request server sized at 500 threads. What breaks, and what do you change?

Each connection holds a thread for its lifetime, so connection 501 waits, and 20,000 threads at ~1 MB of stack would need 20 GB. Move the long-lived connections to an event-loop process, where an idle connection costs kilobytes, and leave request/response traffic on the thread pool.