A request through the stack, hop by hop

Every arrow drawn between two boxes is, underneath, four layers of encapsulation and a dozen routing decisions. This traces one and then prices it: a browser (the client C, at private address C_IP, MAC C_MAC) asking a web app server W at W_IP for the weather in Area A, which in turn calls an API server S at S_IP for the raw data. The layers are covered in the TCP/IP model.

One request across three hops: the packet stays the same, the frame is rebuilt each time100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

Flow 1 — client to web app server

1. Application. The name resolves to W_IP through DNS — which must finish first — and the browser writes an HTTP GET /weather?area=A.
2. Transport. Browsing needs reliability, so the OS picks TCP: source port 51000, destination port 443, sequence numbers. That is a segment.
3. Internet. An IP header goes on top: source C_IP, destination W_IP, written once and final. That is a packet; W_IP is off-LAN, so the routing table hands it to the default gateway.
4. Link. The packet is wrapped in a frame from C_MAC to the gateway's MAC — the next hop, not the destination — and put on the wire.

Every router repeats one loop: strip the link header, read the destination IP, consult its routing table, pick the next hop, build a new frame around the same untouched packet. That asymmetry is the architecture: the packet is addressed once and knows the destination, the frame is rebuilt per link and knows only its two ends. W runs it backwards — frame off, IP header off, port 443 matched to a socket, bytes up as an HTTP GET.

Flows 2 to 4 — the internal call and the replies

W is a client itself here, same machinery with new labels: GET /data?lat=X&long=Y in a segment from port 62000 to port 8080, inside a packet from W_IP to S_IP. Both replies reverse the addresses, and the port W chose in Flow 2 is how its OS knows which socket S's answer belongs to. That four-tuple — source IP, source port, destination IP, destination port — is what keeps the flows apart: W talks to C on port 51000 and to S on port 62000 at the same instant over one card, and nothing above the transport layer thinks about it.

Where the packet does change

"The IP packet travels unchanged" is the textbook version; two things bend it, both as production bugs.

The home router rewrites the envelope. C_IP is private — 192.168.1.20 — and no public router forwards one, so Router 1 performs NAT: the source IP becomes the router's public address, port 51000 becomes a free port of its own, and the pair goes in a table it consults when the reply arrives.

Two consequences follow. First, C_IP never leaves the LAN, so nothing downstream recovers it — not W, not a proxy, not a header. W sees one address shared by every device behind it, and behind a carrier-grade NAT by thousands of unrelated subscribers, so per-IP rate limiting buckets them together. X-Forwarded-For does not undo that: every proxy sits outside the NAT, so what it observes is that same shared address. XFF fixes a different problem — the client address when your own reverse proxy or CDN terminates the connection and would otherwise show its own IP. Key on an authenticated user or API key instead. Second, the NAT entry expires after minutes idle, and the next packet on an evicted entry is dropped silently with no error to either end: how an idle WebSocket dies without a close frame, and why keepalives exist.

The response is not one packet. The Ethernet MTU is 1,500 bytes, leaving a maximum segment size of 1,500 − 20 IP − 20 TCP = 1,460. A 10 KB page is 10,240 ÷ 1,460 = 7.01 — seven full segments plus a 20-byte remainder, so eight packets, each independently droppable. TCP delivers in order, so a lost segment 3 strands 4 through 8 until the retransmission lands: one extra round trip for the page. That is head-of-line blocking, and much of why p99 is worse than p50; see tail latency.

Where the milliseconds go

Every arrow has a price and most are serial. Client in Mumbai, W in US East at roughly 200 ms round trip on the latency ladder, cold connection:

DNS, cold                        ≈ 230 ms
TCP handshake        (1 RTT)       200 ms
TLS 1.3 handshake    (1 RTT)       200 ms
HTTP request + response (1 RTT)    200 ms
                                --------
before the first byte of HTML    ≈ 830 ms

W has not run a line of application code yet. That is why connection reuse is the cheapest latency win available: with the name cached and the connection open, the page costs one 200 ms round trip, not 830.

Flows 2 and 3 are the arrows the client never sees, and their price depends on where S lives. In W's datacenter the round trip is 0.5 ms — 200 ÷ 0.5 = 400× under the client's leg, invisible. If S is a third-party API 80 ms away and W opens a fresh connection per request, the cost is TCP plus TLS plus the request: 3 × 80 = 240 ms, serial, because Flow 4 cannot start until Flow 3 lands. One line on a whiteboard added 240 ms to a budget already at 830, and a pool pays those handshakes once per connection instead of once per request, turning 240 back into 80.

In an interview

What is being tested is whether you know which layer owns a problem, because that decides which component can fix it. They listen for "that is a routing decision, so it belongs to IP and the fix is in the network or the load balancer" rather than a reflexive application change.

Phrasing that lands: "the packet is addressed once end to end, the frame is rebuilt at every hop, so anything reading the destination IP is a router's job and anything reading ports or headers is a proxy's."

The mistake that loses points is treating the arrow between two boxes as free. Candidates draw a service calling three others, quote a 200 ms p99 target, and never notice they have committed to three serial round trips plus handshakes. Say the budget out loud and divide it: 200 ÷ 3 ≈ 66 ms per sequential internal call, including its own database work — which rules out a cross-region hop for any of them.

Check yourself

1. W serves 500 requests per second, each making one serial call to S, which takes 200 ms on average. How large must the pool of connections to S be, and what breaks if it holds 50?

Little's Law: concurrency = arrival rate × mean latency = 500 × 0.2 = 100 in flight. The law needs the mean; sizing against a p99 would be a deliberately conservative headroom choice, not this calculation. A pool of 50 serves half and queues the rest, and that queue time lands on the largest item in the budget: W's p99 crosses its own timeout and W fails requests S is answering correctly. Size near 100 with headroom, cut S's latency, or shed load.

2. You rate-limit at 100 requests per minute per source IP. A university reports that none of its students can use the app. What happened, and what do you key on instead?

The campus sits behind one NAT, so thousands of devices arrive as a single source IP and blow a per-device limit in seconds. C_IP never left the campus LAN and no header recovers it: a trusted proxy's X-Forwarded-For reports that same campus address, because the proxy is outside the NAT too. Key on an authenticated user or API key. With no identity available, per-IP cannot separate one student from the other thousands, so set the ceiling for a shared address, not a device.