Cost control: token budgets, caching and routing
Cap the tokens per task, cost a cached prefix, and price a cascade to a cheaper model for 3,000 tickets a day, knowing which saving needs an eval first.
On this page
An automation's bill is three numbers multiplied: tasks, tokens per task, and price per token. The owner controls all three. The levers that move them most are a per-task budget that bounds the tokens, a cached prefix that cuts the price of the part every call repeats, and a cascade that sends only the harder work to the expensive model. This lesson works all three for support triage at the online retailer from approval gates, not the agency: 3,000 tickets a day.
The assumptions
The course's assumed prices: the large model at $2 per million input tokens and $8 per million output; the small model at one-fifth of that, $0.40 and $1.60; a cached prefix at one-tenth of the input price, $0.20 per million on the large model. Substitute your own; the arithmetic keeps its shape when a price changes. A month is 30 days, so 3,000 tickets a day is 90,000 a month.
The per-task budget
The first version of triage sent the whole ticket thread with every call and paid whatever that cost. Threads grow: a ticket open for three weeks carries forty messages. Input averaged 1,200 tokens of system prompt plus 4,000 of thread:
input 90,000 × 5,200 = 468M at $2 = $936
output 90,000 × 250 = 22.5M at $8 = $180
total $1,116 a month
Nobody chose that number; it was whatever the threads happened to weigh. A per-task budget turns it into a decision. Categorising a ticket and drafting a first reply need the ticket and the last three messages, not three weeks of history, so the variable part is capped at what the task uses:
per ticket system prompt 1,200 tokens identical on every call
ticket + last three messages 1,000 average, capped at 3,000
output: category + draft 250 average, capped at 400
At list price that is 2,200 × $2/M + 250 × $8/M = $0.0044 + $0.0020 = $0.0064 a ticket. The caps are enforced in the harness, not requested in the prompt. A ticket that still exceeds 3,000 tokens after trimming goes to a person, because a single 3,000-token message is rarely routine. A draft that hits the 400-token output cap is incomplete, so it is queued, never sent.
Cost per ticket is the figure for the dashboard, because it does not move with volume. Daily spend rising with tickets is growth. Cost per ticket rising is a change in behaviour.
Baseline: one large model, no cache
per day 3,000 × 2,200 = 6.6M in 3,000 × 250 = 0.75M out
per month 198M in 22.5M out
cost 198 × $2 = $396 22.5 × $8 = $180 total $576
Output is 10% of the tokens and 31% of the bill, because an output token costs four times an input token. No cache ever touches that line.
Lever 1: cache the identical prefix
Only the 1,200-token system prompt is byte-for-byte identical on every call. The ticket differs every time and pays full price whatever the layout. How a cache matches a prefix, and why one timestamp at the top of the prompt turns every call into a cache write, is taught in prompt caching. For triage:
cached prefix 90,000 × 1,200 = 108M at $0.20 = $22
uncached ticket 90,000 × 1,000 = 90M at $2 = $180
output 22.5M at $8 = $180
total $382 a month
That saves $194 a month for changing nothing but the order of the request. The figure treats every prefix as a hit: at 6.25 tickets a minute through the working day the cache entry stays warm, and the few writes after an idle night are noise. The proof is the cache-read counter in the provider's usage data. Here 108M of 198M input tokens should be reads, about 55%; a day near zero means a deploy put something per-request at the top of the prompt.
Caching also makes a longer prompt cheap. Doubling the prefix to 2,400 tokens, to add worked examples for each category, adds 108M cached tokens at $0.20 — $22 a month — where uncached it would add $216. If the examples move the eval score, that is the cheapest accuracy on the sheet.
Lever 2: a cascade to the small model
Not every step needs the large model. Choosing one of 12 categories is a short classification; drafting a reply is the harder work, and only about 30% of tickets end with a reply sent. In the cascade, the small model categorises every ticket, writing about 10 output tokens, and only the 30% that will get a reply go on to the large model for a draft:
small, every ticket 198M in at $0.40 = $79
90,000 × 10 = 0.9M out at $1.60 = $1
large, 30% drafted 27,000 × 1,200 = 32.4M cached at $0.20 = $6
27,000 × 1,000 = 27M in at $2 = $54
27,000 × 250 = 6.75M out at $8 = $54
total about $195 a month
Caching the small model's own prefix would take off roughly $39 more; the figure leaves it out to stay conservative.
The cascade is a third of the capped baseline, and every figure in it depends on something no price list says: how often the small model gets the category wrong. At 3,000 tickets a day, a 4-point drop in category accuracy is 120 tickets a day in the wrong queue. Each costs an agent about 3 minutes to re-route, $1.50 at the assumed $30 an hour, as sampling review prices it: $180 a day, about $5,400 a month, to save $187.
So the cascade waits for the eval set from what working means. Run the 200 cases through the small model, compare it with the large model category by category, and let it decide only the categories where it holds; the rest go to the large model. The router is a classifier with an error rate of its own, taught as a mechanism in model selection and routing. The owner's decision is which categories it may decide alone.
From figures to a budget
Three figures — $576, $382 and about $195 — are forecasts. The monthly budget is set on the one the evidence already supports, with headroom, and watched with an alarm on the daily rate rather than a cap that fires in the last week. Setting that number and placing the alarm is a monthly token budget for triage.
Keep the person-side sum beside it. 3,000 tickets at 2 minutes each is 100 person-hours a day, $3,000 a day at the assumed staff rate. In every configuration above, the token bill is small beside that. The reason to control it anyway is that an uncontrolled bill is the visible sign of an uncontrolled automation.
In practice
When a vendor quotes per seat or per run, ask for cost per ticket at your volume with the token counters visible: input, cached input, output. Ask whether their request layout keeps the shared prefix first. Tell the team the per-task caps, and that a routing change ships only with its eval result beside the saving.
The mistake that costs money is routing to the cheaper model before the eval says what it gets wrong. The $187 a month it saves is less than two days of the re-routing a 4-point accuracy drop costs.
Check yourself
1. The prompt owner wants 1,800 tokens of category examples added to the system prompt, taking it to 3,000. What does that add to the cached monthly figure, and what decides it?
Cached: 90,000 × 3,000 = 270M at $0.20 = $54, against $22 now, so +$32 a month and $414 in total. Uncached, the extra 162M tokens would add $324. The eval decides: if the examples lift category accuracy by even one point — 30 fewer misfiled tickets a day, $45 of re-routing — they pay for themselves many times over.
2. Product wants longer drafts: 500 output tokens instead of 250. How much does each configuration move?
Output is the line caching never touches. Baseline and cached each add 22.5M output tokens at $8, $180: $576 becomes $756 and $382 becomes $562. In the cascade only the 30% drafted pay it, 6.75M more at $8, so $195 becomes about $249. Longer drafts strengthen the cascade's case, but its condition — the eval on the small model's categories — is unchanged.
3. Spend for the week runs 50% over the cached forecast while ticket volume is flat. Where do you look first?
At cost per ticket broken into its counters. If cached input has fallen to near zero, a deploy changed the prefix: the 1,200 tokens now cost $2 per million instead of $0.20, which is +$194 a month, 51% on $382 — what you are seeing. If cached input is normal and input per ticket rose, the cap was lifted or bypassed; if output rose, the draft grew. Each has a different fix, and only the counters say which.