Prompt caching

A prompt cache stores the model's processed state for a prefix of the request, so the next request that starts with the same bytes pays a cache-read rate for those tokens instead of the full input price. The word that matters is prefix. The cache does not find a shared system prompt wherever it sits; it matches from the first byte forward and stops at the first difference, so one timestamp at the top of the system prompt turns a feature that should be mostly cache reads into one that is entirely cache writes.

What the cache matches

The request is rendered in a fixed order — tools, then the system prompt, then the messages — and hashed as one byte sequence. A cache entry covers everything from the start up to a breakpoint you mark (or the vendor marks automatically); a later request hits if its bytes match up to that point. Three prices apply to the tokens involved, all named in the four numbers:

• a token that misses is paid at P_in
• a token written into the cache is paid at C_w × P_in, with C_w a little above 1
• a token read from the cache is paid at C_r × P_in, with C_r well below 1

So a write costs slightly more than no cache at all, and a read costs a fraction of it. Caching pays when reads outnumber writes by enough to cover the surcharge — which is nearly always, once the prefix is stable. Entries expire after a time-to-live measured in minutes, refreshed on each hit, so a feature that calls more often than the TTL keeps its entry warm.

Anthropic's "Prompt caching" reference (platform.claude.com (opens in a new tab)) specifies the ordering, the breakpoints, the minimum cacheable length and the TTL; OpenAI's "Prompt caching" guide (platform.openai.com (opens in a new tab)) describes automatic prefix matching with the same first-byte rule. The multipliers C_w and C_r are read off those pages into the constants lesson, so a price change is one edit.

The byte that invalidates everything

The mistake, in order of how often it appears: a system prompt that opens with "Today is Thursday 16 October 2026, 09:41"; a tool list built from a dictionary whose iteration order differs between workers; a JSON blob serialised with unsorted keys; the customer's name interpolated into the first line of the instructions. Each changes bytes early in the request, and every byte after the change is a miss. The usage counters show cache_creation_input_tokens on every single call and cache_read_input_tokens at zero — the feature is paying C_w × P_in on the whole prefix, every time, which is more than if caching were off.

Expand

The same 36k-token request laid out twice: with the timestamp first nothing before it is stable, with it last the tools and system prompt are one reusable prefix100%−+FitSVG (opens in a new tab)CloseEsc

Pinch to zoom · drag to panScroll to zoom · drag or arrow keys to pan · 0 fits · Esc closes

The fix is an ordering rule. Tools, sorted by name. The system prompt, with nothing per-request in it. Then the conversation, with anything that varies — the date, the user's name, the retrieved documents, the question — in the messages, latest last. Put the breakpoint after the system prompt, and a second one after the retrieved context if the same documents are reused across turns. Nothing about the content changes; only its position.

The worked example

A triage feature carries a 12k-token system prompt and runs 1,000 calls a day, each with about 1k tokens of ticket after the prompt. The ticket differs every call and is paid at P_in either way; the question is the 12k.

Without a hit, the prompt alone is 12k × 1,000 = 12M tokens a day:

no cache   = 12 × P_in per day

With the prefix stable, assume the entry is written once an hour and read for the rest of that hour — 24 writes and 976 reads a day. At 1,000 calls a day the gap between calls is about 86 seconds, inside a TTL of minutes, so the entry stays warm and this assumption is pessimistic:

cached     = 12k × (24 × C_w + 976 × C_r) ÷ 10⁶ × P_in
           = (0.288 × C_w + 11.712 × C_r) × P_in per day

Substitute your own C_w and C_r. With C_w near 1 and C_r a small fraction, the saving is roughly (1 − C_r) of the prompt's cost, and the write surcharge on 24 calls is noise against it. The writes are not noise against what is left: each costs as much as C_w ÷ C_r reads, so the 24 of them are a visible share of the cached bill, which is why the formula keeps both terms. Over a 30-day month the difference is 360 × P_in against (8.64 × C_w + 351.36 × C_r) × P_in.

Now the misconfigured case, with the timestamp at the front. Every call is a write:

miss every call = 12 × C_w × P_in per day

That is worse than no caching at all by the surcharge, every day, and it is the common state of a feature whose owner "turned caching on" and never read the counters. The difference between the two configurations is one line moved from the system prompt to the user message.

Proving it: the usage counters

The response's usage carries three input counters, and their sum is the full prompt:

input_tokens + cache_creation_input_tokens + cache_read_input_tokens = tokens sent

The hit rate for a feature is cache_read ÷ (input + cache_creation + cache_read), computed over a day's calls. For the triage feature above the expected value is 11.7M ÷ 13M ≈ 0.9, since the 1k ticket per call is never cached. Log all three counters per call in the trace, as observability and tracing sets out, and alert when the day's hit rate drops below what the layout predicts: a deploy that added a date to the prompt shows up as the rate falling to zero within the hour, long before the invoice does.

Cost per task, not hit rate, is the number that goes on the dashboard. Hit rate is the diagnostic.

Layout for an agent loop

In a loop the picture is better than for a single call. Every turn's request is the previous turn's request plus one exchange, so the previous request is a prefix of the new one. Move the breakpoint forward to the latest message each turn and the whole history is a cache read; only the newest exchange — the model's last reply and the tool result — is new, and it is written at C_w × P_in. The 365k-token, 20-turn run from where the money goes becomes a 4k write at turn 1 and nineteen 1.5k writes after it, plus reads of everything else: 32.5k × C_w + 332.5k × C_r, against 365k at list price. The quadratic series is still there — it is just priced at C_r.

Two things break this. Compaction rewrites the early history, which changes early bytes and forces a fresh write on the compacted request; that is a cost worth paying once per compaction, not a reason to avoid it. And switching models mid-loop starts from an empty cache, since entries are per model — the point model selection and routing returns to.

In an interview

The question is "how would you make this cheaper", and caching is the expected answer; what is tested is whether you know the mechanism well enough to lay out a request for it. Say "prefix" in the first sentence. Then the ordering: tools, system, messages, stable first, volatile last, breakpoint after the shared part. Then the proof: "I'd read cache_read_input_tokens over a day and expect about 90% for this feature". Then the arithmetic in C_w and C_r, with the write surcharge named.

The mistake that loses points is putting per-request content before the shared content — a date, a user name, a request id at the top of the system prompt — and reporting that caching "didn't help". The interviewer knows the counters would have shown every call as a write.

Check yourself

1. A feature's system prompt is 800 tokens and the retrieved context per call is 15k, different for each user. The owner wants to cache the system prompt. Worth doing?

Only 800 of 15,800 tokens repeat, so the best case saves 0.8k × (1 − C_r) per call out of 15.8k — about 5% of input. Check the minimum cacheable length first; an 800-token prefix may be below it and silently never cache. The lever here is the 15k of context, not the prompt: cache it only if the same user's documents are reused across turns in one session.

2. The day after a deploy, a feature's daily hit rate goes from 0.9 to 0.0 and cache_creation_input_tokens equals the prompt size on every call. What was deployed, and what does it cost until fixed?

Something that varies per request moved into the prefix — a timestamp, an id, a reordered tool list. Every call now writes 12k at C_w × P_in instead of reading at C_r × P_in: on 1,000 calls a day that is 12 × C_w × P_in instead of (0.288 × C_w + 11.712 × C_r) × P_in, up to a factor of C_w ÷ C_r on the prompt's line. Diff the rendered request between two consecutive calls; the first differing byte is the bug.

3. A 30-turn agent compacts its history at turn 15. How many cache writes does the run take, and is the compaction still worth it?

Thirty writes: the 4k prompt at turn 1 and a ~1.5k exchange on each later turn, except that at turn 15 the rewritten history changes every byte after the system prompt and is written again. At turn 15 the request is 4k + 14 × 1.5k = 25k, 21k of it history; compact that to a 3k summary and the large write is ~3k at C_w × P_in, once. Each of the 16 turns from 15 to 30 then reads 18k fewer tokens: 288k × C_r saved against 3k × C_w paid. Compare cost, not tokens — the write is priced at C_w and the reads at C_r — and compaction loses only if one write costs more than 96 reads, so yes.