API Gateways

Once a system is more than one service, every client faces a question with no good answer: which host do I call, and who authenticates me? An API gateway makes that question disappear by presenting a single entry point and handling the cross-cutting work behind it.

An API gateway as the single entry point in front of three services100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

• Definition: An API Gateway is a server or service that acts as a single entry point for client requests destined for backend services, particularly in a microservices architecture. It sits between the clients — web browsers, mobile apps, partner systems — and the collection of backend services, acting as a reverse proxy tailored for API traffic.

• The problem solved: A mobile app that needs a user profile, order history and recommendations for one screen would otherwise call each service itself, know every service's address and port, and carry its own copy of authentication, rate limiting and logging. That makes the client complex, multiplies network round trips, duplicates cross-cutting logic inconsistently across services, and couples the client to the internal service layout, so an internal refactor turns into a client release.

Put a number on the chattiness. A mobile client in India calling a US East region pays roughly 200 ms per round trip. Five sequential calls to assemble one screen is 5 × 200 ms = 1 s of pure network time before anything renders, and that is with every service answering instantly. One call to a gateway that makes those five calls internally pays the 200 ms once and spends at most 5 × 0.5 ms = 2.5 ms inside the datacenter, less when it issues them in parallel. The gateway made nothing faster; it moved the round trips to the side of the wire where they cost 400x less.

What it does

• Request routing to the right service by path, header or method.
• API composition: one client request fans out to several services and their responses merge into one, which is what removes the chattiness above.
• Protocol translation: REST over HTTP facing clients, gRPC facing services.
• Authentication and authorization once at the edge, so services behind it trust a forwarded identity instead of each re-validating a token.
• Rate limiting and throttling per client, key or tier, protecting whatever is behind.
• Caching of backend responses, cutting both latency and load.
• Logging, metrics and tracing in one place, where a request ID is stamped for the rest of its journey.
• TLS termination: decrypt at the edge, speak plain HTTP inside a trusted network, and keep certificate and cipher cost off every service.
• Load balancing across instances of a backend service, often alongside a dedicated load balancer.
• Request and response transformation: formats rewritten and headers added or stripped in flight.

What it buys: one address and one contract for clients, an internal architecture free to be refactored without touching them, cross-cutting logic written once rather than per service, a single choke point for security policy, and caching plus aggregation that can leave the client faster than direct calls would.

What it costs: an extra hop per request, a component that must not fail and so runs as several monitored stateless replicas, one more thing to configure, deploy, scale and be paged for, and — if business logic creeps in — the monolith the services were split to avoid.

What the extra hop costs

The hop is one datacenter round trip, 0.5 ms, plus the gateway's own work — TLS termination, token verification, route match — call it 1–2 ms at the median. Against a 200 ms p99 budget that is at most 1%, and arguing about it wastes interview time.

Aggregation is where the cost stops being trivial, because a composed response cannot return until its slowest branch does. Four backend calls in parallel, each meeting 120 ms 99% of the time, produce an aggregate that meets it only 0.99^4 = 96.1% of the time: one response in twenty-five misses, not one in a hundred. Restoring a true aggregate p99 needs every backend at p99.75, since 0.9975^4 ≈ 0.99 — a far more expensive service to build. The remedies are the ones from tail latency: shrink the fan-out, hedge, or return a partial response with the slow section marked pending. Adding gateway replicas does nothing here, because the tail is downstream.

Sizing it, and the failure that takes everything down

A gateway holds a connection open for the whole downstream call, so Little's Law sizes it: concurrency = arrival rate × latency. At 200M requests/day — 200M ÷ 86,400 ≈ 2,300 QPS average, about 7,000 QPS at a 3x peak — and 150 ms per request, that is 7,000 × 0.15 = 1,050 requests in flight: sockets, buffers, and in a thread-per-request runtime, threads.

Now one route carrying a fifth of the traffic degrades from 150 ms to 2 s. Arrival rate has not changed, but in-flight demand becomes 5,600 × 0.15 = 840 for the healthy routes plus 1,400 × 2 = 2,800 for the sick one, so 3,640 against the 1,050 provisioned. The pool exhausts, and requests to perfectly healthy services queue behind requests to the broken one. The gateway does not fail because it is slow; it fails because it is shared.

The fix is a per-route concurrency cap — a bulkhead set near that route's normal in-flight number — so a failing route sheds load with a 503 and cannot reach past its own budget. Pair it with a gateway timeout below the client's and a circuit breaker, so calls to a route already known to be failing release their slot in milliseconds instead of holding it for 2 s. The rejected alternative, raising the global connection limit, buys a few minutes and then collapses with a longer queue behind it.

Retries are a contract, not a checkbox

Gateways offer "retry on 5xx or timeout" as one line of config. Enabled globally with two retries — three attempts in all — a backend failing half its requests receives 1 + 0.5 + 0.25 = 1.75x its normal load at the moment it can least afford it. Cap retries as a share of traffic — a 10% retry budget, refused once exceeded — so a partial outage cannot feed itself into a full one.

The subtler problem is that a timeout is not a failure, it is ambiguity: the write may have committed and only the response been lost. Retrying a POST at the gateway makes delivery at-least-once, at-least-once means duplicates, and duplicates mean the handler must be idempotent. Retry idempotent methods freely; retry a write only when the client supplied an idempotency key that the gateway forwards unchanged.

One gateway, or one per client

Mobile wants a compact screen payload, the partner API wants stable versioned resources, the web app wants something in between. One gateway serving all three accumulates per-client transformation rules until the god-gateway problem arrives wearing a config file. Backend-for-frontend splits it: one gateway per client type, each owned by the team that owns that client, over a shared auth and rate-limit layer. It costs three deployments, three configs and three on-call rotations, so take it only once payloads have actually diverged — two clients wanting the same JSON need one gateway, not two.

Keep east-west traffic out of it either way. The gateway exists for requests entering from outside; internal service-to-service calls should go direct or through a mesh sidecar. Routing them through the edge pays the hop twice and makes an internal path depend on the one component every external client already depends on.

Off-the-shelf options: AWS API Gateway, Google Cloud API Gateway and Apigee, and Azure API Management among managed services; Kong Gateway, Traefik, Spring Cloud Gateway and Ocelot if self-hosted.

In an interview

What is being tested is whether you can place cross-cutting concerns once instead of N times, and whether you notice, unprompted, that you have just drawn the box that takes down every API. Introduce it as a façade and price it in the same breath: "Three client types hit one gateway that terminates TLS, validates the token once, applies per-key rate limits, and routes. Services behind it are not reachable from outside and trust the forwarded identity header. It adds 1–2 ms at the median and becomes the thing that must not go down, so it runs as stateless replicas behind a load balancer, with a per-route concurrency cap so one slow service cannot eat the pool." Routing, auth, rate limiting and aggregation show you know the functions; the concurrency cap shows you have operated one.

The mistake that loses the most points is treating the gateway as free, infallible infrastructure — one box on the diagram, no replica count, no timeout, and no answer to "what happens when the recommendations service takes 2 s?" It turns a resilience discussion into an availability hole you drew yourself. The next most expensive mistake is business logic in the gateway: order totals, entitlement rules, anything that has to be redeployed when a product rule changes. Cross-cutting and stateless stays, domain logic goes to the service that owns the data. Propose a gateway when external clients face several backend services; skip it for a single service, where it is one more thing to run.

Check yourself

1. Your end-to-end p99 budget is 200 ms for users in India, served from US East. The gateway aggregates 4 backend calls in parallel, each with a p99 of 120 ms. Does the design meet the budget, and what do you change?

No, and no backend work will fix it: the India to US East round trip alone is roughly 200 ms, so the budget is gone before the gateway is reached. Server-side the design is close to fine — the fan-out meets 120 ms 0.99^4 = 96.1% of the time, so the aggregate p99 lands somewhat above 120 ms, plus 1–2 ms of gateway. The decision is geographic, not architectural: terminate closer to the user with a regional gateway or an edge CDN tier, or restate the target as a server-side p99, which is what most teams actually measure.

2. Peak traffic is 7,000 QPS with downstream p99 at 150 ms. One route carrying a fifth of it degrades to 2 s. How much is in flight, and what is the first change you make?

Normally 7,000 × 0.15 = 1,050 in flight. After the degradation, 5,600 × 0.15 = 840 plus 1,400 × 2 = 2,800, so 3,640 — about 3.5x what the gateway is sized for, which exhausts the pool and stalls healthy routes. First change: a per-route concurrency cap set at that route's normal in-flight number, 1,400 × 0.15 = 210, which serves 210 ÷ 2 s = 105 QPS of it and rejects the rest with a 503. Total in flight then lands at 840 + 210 = 1,050, exactly what the gateway is provisioned for, so the other four-fifths keep their latency. Then a timeout under the client's and a circuit breaker to free slots quickly. Not a bigger global pool, which only enlarges the queue and delays the same collapse.

3. Two internal services exchange 3,000 requests per second. Do you route them through the gateway?

No, and the reason is not capacity. Little's Law takes the whole downstream call, not the gateway hop: a 20 ms internal call puts 3,000 × 0.02 = 60 requests in flight, a rounding error against the 1,050-slot pool — though at a 150 ms call it would be 3,000 × 0.15 = 450, 43% of that pool and no longer nothing. The reason is that the internal path would then depend on the edge. Availability multiplies along a path, so a 99.99% service reached through a 99.99% gateway gives about 99.98%, taking expected downtime for that path from roughly 4.3 minutes a month to 8.6, and any gateway incident now breaks traffic that never left the datacenter. Keep east-west direct or on a mesh; the gateway earns its hop on north-south traffic, where the auth, rate limiting and single address pay for it.