Cost, latency and routing · Cost and caching

Where the money goes

Cost an agent loop as a loop rather than as one call, build the per-feature cost sheet, and pick the lever that cuts the bill first.

Lesson · 8 min · 22 of 77

On this page

A model bill has two lines, input tokens and output tokens, and almost every surprise on it comes from the first. A single call reads a few thousand tokens and writes a few hundred; a 20-turn agent reads its whole history again on every turn, so it reads an order of magnitude more than it writes before anyone notices. Costing a feature means counting what is sent, not what comes back.

The formula and the two counters

Every call returns a usage block with input_tokens and output_tokens. The bill is those two counters, summed over every call, priced with the course's named constants from the four numbers: P_in is the price of a million input tokens, P_out the price of a million output tokens.

cost of one call = (input_tokens ÷ 10⁶) × P_in + (output_tokens ÷ 10⁶) × P_out

P_out is a multiple of P_in, so a naive reading says output is the expensive side. For a single call that is roughly true: 3,000 tokens in and 600 out is 0.003 × P_in + 0.0006 × P_out, and with P_out at several times P_in the two terms are close. The arithmetic changes shape the moment there is a loop.

The loop resends the history

The mistake is costing an agent as "one turn, times the number of turns". Take a tool-using agent with a 4k-token system prompt, where each turn adds about 1.5k tokens to the conversation — a 300-token tool call, a 1,000-token tool result, 200 tokens of text — and the model writes about 500 tokens a turn including the call. Turn 1 sends 4k. Turn 2 sends 4k plus turn 1, so 5.5k. Turn n sends 4k + 1.5k × (n − 1), and turn 20 sends 32.5k.

The history is resent every turn, so input grows linearly and the sum grows with the square of the turn count.

Summed over the 20 turns:

input  = 20 × 4k + 1.5k × (0 + 1 + … + 19) = 80k + 1.5k × 190 = 365k tokens
output = 20 × 500 = 10k tokens
cost   = 0.365 × P_in + 0.010 × P_out

The loop reads 36 tokens for every one it writes. Even with P_out at five times P_in, the output term is 0.05 × P_in against 0.365 × P_in for input: input is seven-eighths of the bill. Double the turns and the input sum roughly quadruples, because the series is quadratic in the turn count. This is the arithmetic behind memory and context management: dropping old tool results is not tidiness: the resent tool results alone are 1k × 190 = 190k of the 365k, the largest line on the invoice.

The counters prove it. Log input_tokens per call in the trace, as observability and tracing describes, and the per-turn number climbing in a straight line is the signature of a loop that keeps everything.

The per-feature cost sheet

A feature is costed on one sheet with the same columns every time: calls per task, input tokens per task, output tokens per task, tasks per day. Multiply through to tokens per month, then price with the constants. Assume 30 days a month.

Feature	Calls/task	Input/task	Output/task	Tasks/day	Input/month	Output/month
Ticket triage	1	6k	300	2,000	360M	18M
Contract summary	1	29k	550	300	261M	5M
Refund agent	8	59k	4k	400	708M	48M

The refund agent is the eight-turn version of the loop above: 8 × 4k + 1.5k × 28 = 74k in with a 4k prompt, trimmed to 59k by dropping tool results older than two turns (turns 4 to 8 drop 1, 2, 3, 4 and 5 of them, 15k), and 8 × 500 = 4k out. In money:

triage   = 360 × P_in + 18 × P_out
summary  = 261 × P_in +  5 × P_out
refunds  = 708 × P_in + 48 × P_out

Three things the sheet makes visible that a dashboard total does not. The refund agent is the biggest line despite the fewest tasks, because it loops. The summary feature's input is almost all document, so prompt caching cannot help it — only the 2k of shared instructions repeats. And triage, the feature everyone assumed was free because each call is small, reads 360M tokens a month because it runs 2,000 times a day. Volume multiplies small calls into large bills; loops multiply large calls into larger ones.

Batch endpoints for work that can wait

Most vendors run a batch endpoint: submit a file of requests, collect the results within a window measured in hours, and pay a lower rate for the input and output tokens in exchange for the wait. Anthropic documents the mechanism and its completion window under "Batch processing" (platform.claude.com (opens in a new tab)); OpenAI's equivalent is the "Batch API" guide (platform.openai.com (opens in a new tab)). The discount is a multiplier on P_in and P_out, stated on those pages, not a different formula.

The question is how long the result can wait. The contract summary is the case that looks like a batch and is not: the reviewer opens the file within 30 minutes of upload, and a window measured in hours cannot promise that, so it runs in the background at upload at list price. Nightly eval runs, backfills over last quarter's tickets, re-embedding a corpus after a chunking change: batch them all. Ticket triage has an agent waiting on the queue, and the refund agent has a customer in the chat; neither can take a window of hours. The rule: if the p95 latency budget in latency budgets and streaming is written in hours, the request belongs in a batch.

The levers, in order

Input first, because input dominates. Each lever is sized by the sheet before it is pulled.

Shorten what repeats. A 6k system prompt sent 2,000 times a day is 12M tokens a day; a 2k one is 4M. Move procedures out of the prompt and into tools or skills that load on demand.
Stop resending old tool results. Compaction or context editing turns the quadratic series back toward a line. For the eight-turn agent it took 74k to 59k, a fifth of the input.
Cache the stable prefix. The tools and system prompt are identical across calls; a cache read costs C_r × P_in instead of P_in. The next lesson does the arithmetic.
Route. Run the eval on a cheaper tier and send it the cases it passes, as model selection and routing describes.
Batch anything whose latency budget is written in hours.

Output has two levers: cap max_tokens at what the consumer uses, and ask for structured fields rather than prose. A 550-token summary that the reviewer reads is worth its P_out; a 400-token preamble before a 20-token classification is not.

The same list, from the operator's side rather than the engineer's, is in cost control: token budgets, caching and routing.

In an interview

The question is usually "estimate what this feature costs a month", and what is tested is whether you cost a loop as a loop. Say the four numbers first — calls per task, tokens in, tokens out, tasks per day — then show the history growing: "turn n resends everything before it, so a 20-turn run reads about 365k tokens, not 20 × 4k". Price input and output separately, name which dominates, and name the lever that follows from it.

The mistake that loses points is multiplying one turn's cost by the number of turns. It understates an agent's input by an order of magnitude, and an interviewer who has run one knows it. The second mistake is reaching for a cheaper model before shortening the prompt: routing changes the price of every token, trimming removes tokens, and removal is the larger saving on a loop.

Check yourself

1. A support agent runs a 6k system prompt, adds 2k of history a turn, and averages 12 turns per task at 500 tasks a day. Input tokens a month, and the first lever you pull?

Turn n sends 6k + 2k × (n − 1); over 12 turns that is 12 × 6k + 2k × 66 = 72k + 132k = 204k per task. 500 a day × 30 days = 15,000 tasks → ~3.1 billion input tokens a month, 3,060 × P_in. The history term (132k) is bigger than the prompt term (72k), so drop old tool results first; the prompt is the second lever, caching the third.

2. Product wants nightly summaries of every ticket closed that day — 3,000 tickets at 5k tokens in and 300 out. One call per ticket as they close, or a batch at midnight?

Nobody reads them until morning, so the latency budget is hours: batch. Daily tokens are 15M in and 0.9M out, so a month is 450 × P_in + 27 × P_out at list price and the batch multiplier scales both terms down. The reverse decision is right only if an agent needs the summary while the ticket is still open, which is a different feature.

3. Two features cost the same per day. One runs 50 times a day in one call, the other runs 5 times a day in ten turns. Which gets the caching work?

The ten-turn one. In a loop the whole history becomes next turn's prefix, so nearly every token after turn 1 is a cache read; in the single call only the shared instructions repeat. Same bill today, but caching cuts the loop's input toward C_r of list price and barely moves the other.