Running it: evals, cost and security · Cost

A monthly token budget for triage

Cost triage at 3,000 tickets a day with and without caching and routing, set the monthly budget line, and place an alarm that fires before the month is lost.

estimatemediumExercise · 4 min · 61 of 75

On this page

Triage at the online retailer, not the agency, goes live next week, and the retailer's founder needs one number for the budget line and one alarm that fires while there is still time to act. The arithmetic is in cost control: token budgets, caching and routing; the decision is which figure to trust.

The situation

Support triage reads 3,000 tickets a day. Every call carries a 1,200-token system prompt, identical on every call, then the ticket plus its last three messages, 1,000 tokens on average. The output — one of 12 categories and a draft reply — averages 250 tokens. About 30% of tickets get a reply sent; the rest need only a category. A misfiled ticket costs an agent 3 minutes to re-route. If the small model only categorises, it writes about 10 output tokens. A reviewer reads 1 sent reply in 20. Nobody has yet run the small model against the 200-case eval set.

Assume the large model at $2 per million input tokens and $8 per million output; the small model at one-fifth of that, $0.40 and $1.60; a cached prefix at one-tenth of the input price, $0.20 per million on the large model. A month is 30 days. A person triaging by hand takes about 2 minutes a ticket, at an assumed staff cost of $30 an hour.

The question

Set the monthly budget, name the two levers you pull first, and say where the alarm sits.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Per ticket, 2,200 in and 250 out; per day, 6.6M in and 0.75M out; per month, 198M in and 22.5M out. Baseline: 198 × $2 + 22.5 × $8 = $396 + $180 = $576.

Lever 1, cache the 1,200-token prefix, the only identical part: 108M cached at $0.20 is $22, 90M uncached at $2 is $180, output $180. $382.

Lever 2, route. The small model categorises every ticket: 198M in at $0.40 is $79, 0.9M out at $1.60 is $1. Only the 30% needing a draft reach the large model: 32.4M cached at $0.20 is $6, 27M uncached at $2 is $54, 6.75M out at $8 is $54. About $195.

Budget $450: the cached figure plus 18% headroom. Not the routed figure, because routing needs the eval set before anyone knows what the small model gets wrong. Ship caching now; route only after the 200 cases show the small model holding category by category.

The alarm is a rate. Expected spend is $382 ÷ 30 ≈ $12.70 a day, so alarm on any day over $16, or on month-to-date running 20% ahead of the pro-rated line. The $450 is a hard stop, not the alarm: a cap alone fires in the last week whether or not anything is wrong.

Against 100 person-hours a day, about $3,000, the budget is small; the alarm is the point. Evidence that changes it: a cache-read share well under 55% of input, which means the layout leaks, or an eval that clears the small model — then rebudget near $230.

Rubric

works per ticket, then per day or month, with the token counts and the arithmetic shown
prices input and output separately, each at its own rate
applies the cached rate only to the 1,200-token identical prefix, never to the ticket
sets the budget from the cached figure and makes routing conditional on the eval set
places the alarm on a daily rate or a pro-rated line, with the cap as a hard stop
compares the budget with the person-hours the automation replaces

Where answers go wrong

"Cheapest model for everything: about $115 a month." No accuracy figure. At 3,000 tickets a day, a 4-point drop in category accuracy is 120 tickets a day in the wrong queue; at 3 minutes each, $1.50, that is $180 a day. Two days of it cost more than the $267 a month saved against the cached figure.

"Cache the whole request: 198M at $0.20 plus $180 of output, about $220." The ticket differs on every call, so only the identical prefix can be read from cache. The forecast is $162 short of the real $382, and the budget built on it is breached in the third week.

"Budget $576, with an alert when spend reaches it." A cap-only alarm fires in the last days of the month, after weeks of overspend. Worse, a cache that silently stops hitting costs exactly $576 — so this budget would never notice the most common cost failure there is. Prompt caching shows how one timestamp at the top of the prompt causes it.