Cost, latency and routing · Cost and caching

A cache layout for the support bot

Reorder a support bot's request so its 12k shared prefix is served from cache, place the breakpoints, and cost the prefix per day before and after.

designmediumExercise · 4 min · 27 of 77

On this page

A support bot's owner turned prompt caching on a week ago and the bill went up. The request has everything the cache needs except the right order.

The situation

The bot answers customer chat for a subscription product. Each call carries 3k tokens of tool definitions, a 9k-token instruction block, and on average 8k tokens of conversation history, then the customer's latest message. The instruction block opens with a line the template fills in per call: Customer: Priya S · Plan: Pro · Now: 2026-10-16 09:41:07. Tools are rendered from a dictionary and arrive in whichever order the worker iterates them.

Volume is 20,000 calls a day. Caching was enabled with one breakpoint at the end of the instruction block; the dashboard shows cache_creation_input_tokens equal to the full 12k on every call and cache_read_input_tokens at zero.

Assume P_in = $3 per million input tokens, a cache write at C_w × P_in and a cache read at C_r × P_in, with C_w = 1.25 and C_r = 0.1, and a cache time-to-live of five minutes, refreshed on every hit. Ignore output tokens and the customer's own message; they are the same under every layout.

The question

Lay out the request so the 12k of shared content is served from cache, say where the breakpoints go, and give the daily cost of that 12k before and after — including what the current misconfiguration costs against caching turned off.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Today every call writes, because the timestamp is the first thing after the tools. Cost of the 12k prefix: 12k × 20,000 = 240M tokens a day at 1.25 × $3/M = $900 a day. With caching off it would be 240M × $3/M = $720. The owner is paying $180 a day for a cache that never hits.

Layout, in render order. Tools first, sorted by name so every worker renders the same bytes. Then the 9k instruction block with the per-customer line removed. Then the messages: the first user message carries Customer: Priya S · Plan: Pro, fixed for the conversation, then the history, then the latest message with the Now: timestamp attached, since a timestamp anywhere earlier would turn the history after it into writes. Breakpoint one after the instruction block; breakpoint two at the latest message, moved forward each turn so the conversation's own history is a read on the next turn, as prompt caching lays out.

Cost after. 20,000 calls a day is one every 4.3 seconds, well inside a five-minute TTL, so the entry stays warm; assume one write an hour to be safe. Writes: 24 × 12k × 1.25 × $3/M ≈ $1.08. Reads: 19,976 × 12k × 0.1 × $3/M ≈ $71.91. Total about $73 a day against $900: $2,200 a month against $27,000. The second breakpoint turns most of the 8k history into reads as well, roughly 160M × 0.1 × $3/M ≈ $48 a day instead of $480.

Verification: after deploy, the day's hit rate cache_read ÷ (input + creation + read) should sit near 0.9. If it stays low, diff two consecutive rendered requests; the first differing byte is the remaining leak.

Rubric

names the timestamp line as the byte that invalidates the prefix on every call
orders tools, then static system, then per-customer and volatile content in the messages, with tools sorted
places a breakpoint after the shared block and a moving one at the latest message
costs the prefix per day before and after with the write and read multipliers shown
states that the misconfigured cache costs more than no cache, with the figure
names the counter and the expected hit rate that proves the layout works

Where answers go wrong

"Move the timestamp to the end of the instruction block." The prefix is now stable up to the timestamp, but the breakpoint sits after it, so the entry still includes a byte that changes per call and every call still writes. Volatile content goes after the breakpoint, not merely later in the same block.

"Cache the history too by putting a breakpoint on every message." Breakpoints are limited per request and each costs nothing to place, but the answer never says what the write-versus-read arithmetic is, so the reviewer cannot tell if the candidate knows a write costs more than list price. The layout is right; the costing is missing.

"$720 a day before, $72 after — a 90% saving." The before figure is caching-off, not the deployed state. The deployed state is $900, and the point of the exercise is that a cache configured wrong is worse than none. Missing that costs the fifth rubric line.