Retries, timeouts and rate limits

A model API fails in four ways that look alike from the caller's side — a rate limit, an overloaded backend, a dropped connection, a request the server rejected — and only three of them deserve a second attempt. A retry policy is the small piece of code that tells them apart, waits the right amount, and gives up at the right time. Written badly it doubles the bill during an outage; written well it is twenty lines, and the code exercise in this module writes them.

Which status codes to retry

A retry re-sends the same bytes. That only helps when the failure was about the server's state, not the request's content:

The never row is the one that gets retried in practice. A 400 says the request body is wrong — a tool schema that fails validation, a message list that alternates roles incorrectly, a max_tokens beyond the model's cap — and sending it four more times with increasing waits produces four more 400s and 15 seconds of backoff (1 + 2 + 4 + 8 on the schedule below) before the user sees the error. The trace shows it as a single call with five attempts and no change in the response; the fix is a status check before the retry loop, not inside it. Anthropic's "Errors" page (platform.claude.com (opens in a new tab)) lists the codes and which are transient, and the SDKs retry the transient set by default — a default worth reading before adding a second retry layer on top of it, which multiplies attempts.

Exponential backoff with jitter

The wait before retry k doubles each time, from a base, up to a cap:

wait_k = min(cap, base × 2^(k − 1))        base 1 s, cap 30 s → 1, 2, 4, 8, 16, 30, 30 …

Doubling matters because the failure that caused the retry is usually overload, and a retry adds load. The mistake is the fixed interval: 200 workers that all hit a 529 at the same instant all retry one second later, in one burst, and the backend that was overloaded by 200 requests now gets 200 more at once, then 200 more a second after. Doubling spreads the attempts in time; jitter spreads them across workers. With full jitter the wait is a random value between zero and wait_k, so the 200 retries land across the whole second instead of at its end.

Expand

A request retried twice with exponential backoff and a retry-after header, then the 400 that ends the attempt instead100%−+FitSVG (opens in a new tab)CloseEsc

Pinch to zoom · drag to panScroll to zoom · drag or arrow keys to pan · 0 fits · Esc closes

Where the server sends retry-after, the wait is at least that long, even above the cap: the header is the server's own estimate of when capacity returns, and retrying earlier only earns another 429, so jitter goes on top of the header, never below it. Cap the attempts as well as the waits — five attempts, 15 seconds of backoff before jitter on the schedule above, is a common shape — and record which attempt succeeded in the trace, because a feature whose calls routinely succeed on attempt 3 is running at a rate limit it has not noticed.

Keep the schedule a pure function: inputs in, a list of waits out. The code exercise retry-with-backoff policy does exactly that, with jitter applied by the caller so the function stays testable.

A timeout scales with the output cap

A timeout is a promise about how long a call may take, and the call's length is set by its output. At the course's decode speed of 50 tokens/s, from the four numbers, a request with max_tokens 4,000 can legitimately run for 80 seconds of decode plus prefill; a 30-second timeout on it does not catch failures, it kills healthy calls about 1,500 tokens into 4,000, and you pay for every token generated before the cut. Derive the number instead:

timeout = first-token allowance + max_tokens ÷ 50 + margin
        = 5 s + 80 s + 10 s ≈ 95 s   for max_tokens 4,000
        = 5 s +  4 s +  5 s ≈ 14 s   for max_tokens 200

Streaming changes the shape of the timeout from one deadline into an idle deadline: abort if no chunk has arrived for, say, 20 seconds. A healthy stream never trips it however long the output, and a stalled one is detected in 20 seconds rather than 95. For long outputs this is the only workable timeout, which is one more reason latency budgets and streaming streams long outputs whether or not anyone is watching.

A timed-out request may have been served. The tokens were generated and billed whether or not the response was read, which is why a timeout followed by a retry is the expensive case: the second attempt pays the full price again, and if the first one had side effects it repeats them.

Side effects need an idempotency key

The model call itself is read-only: retrying it costs money, not correctness. The tool calls the model makes are not. An agent that calls issue_refund and loses the response cannot tell a refund that never happened from one whose confirmation was dropped, and the retry issues it twice. This is the chain from idempotency: at-least-once delivery produces duplicates, so duplicates must be harmless — a key on every side-effecting call, generated once per logical action and sent on every attempt, with the tool returning the stored result on a replay.

In the agent loop the key comes from the harness, not the model: the tool-call id the model emitted is a reasonable key for one turn, since a retry of that turn replays the same id. It is one of the reasons reliability lives in the harness — a prompt line "do not refund twice" cannot survive a lost response, and a key does. The founder-facing treatment, with the connectors that need it, is idempotency and deduplication.

Rate limits and Little's law

Vendors limit by requests per minute and tokens per minute per tier; Anthropic documents its tiers and the 429 behaviour under "Rate limits" (platform.claude.com (opens in a new tab)). Before asking whether a feature fits its limit, work out how many calls it holds open at once, because that is what a connection pool, a worker count and a semaphore are sized from. Little's law gives it directly: concurrency equals arrival rate times time in the system.

50 tasks/s × 8 s per task = 400 tasks in flight

Four hundred open requests means a pool of at least 400 connections, and a semaphore of 400 around the client, not the default of a few dozen. The same law explains why retries during an outage are so damaging: an overloaded backend raises the time per call, concurrency rises with it, the pool fills, and retries raise the arrival rate on top — both terms of the product moving the wrong way at once, which is the cascade the system-design lesson walks through.

Token throughput follows: 50 tasks/s with 4 calls of 8k tokens each is 50 × 4 × 8,000 × 60 ≈ 96M tokens a minute. Compare that to the tier's token limit. Over it, retries are the wrong tool — the 429s are not transient, they are the limit — and the right one is a queue in front of the client that admits work at the limit's rate, the token-bucket shape from rate limiting pointed at your own outbound traffic. The small-company version of this lesson is rate limits, retries and backoff.

In an interview

The question is "the model API starts returning errors at peak — what does your client do?", and what is tested is whether you separate the cases. Say which codes retry and which do not, with the 400 named as the one that must not. Give the schedule — "doubling from one second, capped at 30, with jitter, five attempts, retry-after as the floor" — and derive the timeout from max_tokens rather than quoting one. Then the key: "any tool with a side effect carries an idempotency key, because a lost response is indistinguishable from a lost request". Then concurrency from Little's law.

The mistake that loses points is retrying a 400, or retrying a side-effecting tool call without a key. Both are the same error: treating a retry as free. A close third is a fixed 30-second timeout on a request whose output cap implies 80 seconds of decode.

Check yourself

1. A feature sets max_tokens 2,000, streams nothing, and times out at 20 s. The trace shows 15% of calls timing out and being retried. What is happening, and what is the fix?

2,000 tokens at 50/s is 40 s of decode, so any answer past ~900 tokens cannot finish in 20 s; the 15% are the long answers, killed and paid for, then paid for again on retry. Either derive the timeout — about 50 s — or stream and use a 20 s idle timeout. Cutting max_tokens is right only if the consumer does not need the long answers.

2. 20 tasks/s, each task 4 calls averaging 3 s, run through a client with a 50-connection pool. Does it hold?

Each task holds a connection for about 12 s, so concurrency is 20 × 12 = 240 against 50 slots: no. Either raise the pool to ~300 with headroom or admit tasks at a rate the pool supports, 50 ÷ 12 ≈ 4 tasks/s. If the vendor's request limit is below 20 × 4 × 60 = 4,800 requests a minute, the queue is needed anyway.

3. An agent's send_email tool lost its response and the harness retried the turn. Two emails went out. Which line of which component is at fault?

Not the prompt and not the model: the tool. send_email accepted a second call with the same tool-call id and sent again, instead of keying on the id and returning the stored result. The fix is in the tool's handler, and the test is a replay of the same id asserting one delivery.