The four numbers
Write the four numbers for any model feature — tokens in, tokens out, calls per task, p95 latency — in the named units every later lesson computes with.
On this page
Every feature in this course is sized by the same four numbers before anything else is said about it: tokens in, tokens out, calls per task, and p95 latency. Cost is the first two multiplied by a price; a loop multiplies both by the third; the fourth is what the user feels. A team that cannot write the four numbers for a feature cannot say what it costs or whether it is too slow, and most launch surprises trace to one of the four having been guessed.
The assumption table
Every lesson after this one does its arithmetic with these names. This page is the only lesson in the course that states a figure for any of them.
| Name | Assumption | Used for |
|---|---|---|
| a token | about 4 characters, or ¾ of an English word — 1.3 tokens per word | every count on every page |
| the window | 200k tokens, one budget shared by system prompt, history, tool results and output | every window budget |
| time to first token | about 1 s for a few thousand tokens in, longer as input grows | every latency estimate |
| decode speed | 50 tokens/s | every latency estimate |
P_in, P_out | the input and output price per million tokens | every cost |
C_w, C_r | multipliers on P_in for writing a prefix to the cache and for reading it back | every cached prefix; module 3 teaches them |
These are assumptions, not measurements. Last checked: 2026-10-16. For the worked figures on this page, assume P_in = $3 and P_out = $15 per million tokens, C_w = 1.25 and C_r = 0.1 — the illustrative values the course's exercises restate when they need a sum. Replace them with your contract's figures; the arithmetic survives, and because the other lessons use only the names, a price change is one edit here and no two lessons disagree.
Number 1 — tokens in
Tokens in is everything the model reads on one call: the system prompt, the conversation so far, every tool result, and the user's text last of all. The user's text is usually the smallest part. A 300-word ticket is 400 tokens; the system prompt around it is 3k; the order record a tool fetched is 500; by the third turn the history carries all of it again.
To count, multiply words by 1.3 or divide characters by 4. A 40-page contract at 500 words a page is 20,000 words, about 26,000 to 27,000 tokens, and fits a 200k window with room; tokens and the context window covers the counter and what happens at the edge. The window is a budget, not a limit you discover: one bar, and everything you put in it is space the output cannot have.
The mistake is to size a feature by the user's text. A team that budgets 400 tokens in for "a 400-token question" has missed the 3k system prompt, the 2k of retrieved policy and the history, and is off by a factor of thirteen on the first turn — 5.4k against 400 — and more on every turn after.
Number 2 — tokens out
Output is what the model writes, and the number is set by what you ask for. A label is 1 to 5 tokens; a label with a reason is 60; a 400-word summary is about 550; a drafted reply is 250 to 300. Output costs more per token — P_out is 5× P_in in our assumption — and, more important, it costs time: at 50 tokens/s, 550 tokens is 11 s of decoding, whatever the input was. "Return the label" against "return the label and explain it" is a 12× difference or more in this number and most of the latency the user will see.
Number 3 — calls per task
A task is what the user asked for; a call is one request to the model. A classifier is one call per task. An agent with tools is three to forty. The number multiplies the other two, and worse than linearly, because every call resends the history: a 20-turn loop whose context grows by 1k a turn reads 4k + 5k + … + 23k ≈ 270k tokens to produce perhaps 4k. Where the money goes works this through. For now, the rule is that calls per task is the number to write down first when a feature has a loop, and the stop conditions are what bound it.
Number 4 — p95 latency
Latency has two parts with different causes. Time to first token grows with tokens in, because the model reads everything before it writes; for a few thousand tokens of input, assume about one second. Decode time is tokens out ÷ 50: 300 tokens is six seconds more. Two calls with a tool lookup between them is a 10-second task.
We quote p95, not the mean, because the mean hides the tail and the tail is what the user sees; tail latency is the same argument for any service. A budget is set against p95, and the design then chooses between streaming, running the task in the background, and cutting tokens out — latency budgets and streaming makes that choice.
The four numbers for one feature
A reply drafter for a support team: two calls per ticket. The classify call carries a 3k system prompt, the 400-token ticket and a 500-token order record. The draft call carries the same, plus 300 tokens of history and about 1k of retrieved policy.
- tokens in ≈ 3.9k + 5.2k = 9.1k
- tokens out = 60 + 300 = 360
- calls per task = 2
- p95 ≈ 1 + 1.2 + 0.6 + 1 + 6 ≈ 10 s
Cost per ticket = 9,100 × $3 ÷ 1,000,000 + 360 × $15 ÷ 1,000,000 = $0.0273 + $0.0054 ≈ $0.033. At 2,000 tickets a day that is $65 a day, about $2,000 a month — and 83% of it is input, the number the team guessed smallest.
The 3k system prompt is identical on every call, so it is a cache candidate. A read of a cached prefix costs C_r × P_in instead of P_in: 3,000 × $3 ÷ 1,000,000 × (1 − 0.1) ≈ $0.008 saved per call, $0.016 per ticket, close to half the bill. The write at C_w = 1.25 recurs only when the entry has expired, which at 4,000 calls a day, one every 22 seconds, is rare. Prompt caching says what has to be true of the prefix for the hit to happen.
The latency decision follows from the fourth number. Ten seconds at p95 is too long for a support rep waiting on a blank reply box, so either the draft streams as it is written or the task runs when the ticket arrives, before anyone opens it. Neither choice exists until the number is on the page.
In an interview
"Size it" is the first of the six question families in what an interviewer asks an AI engineer. Say the four numbers with the arithmetic shown — "about 9k in, 360 out, two calls, 10 s at p95; three cents a ticket, two thousand a month, 83% of it input" — then name the levers: caching the fixed prefix halves the bill, and streaming the draft puts its first words in front of the rep at about 4 s instead of 10.
The mistake that loses points is pricing on tokens out. Output carries the higher price, so candidates size it and ignore input; in a loop, input volume is 20 to 70× output and dominates the bill. The second mistake is quoting a mean: a 4-second mean with a 12-second p95 is a 12-second feature.
Check yourself
1. A summary feature: 400-word summaries of 27k-token documents, 300 a day, read by a reviewer who opens the file within 30 minutes of upload. Where does the money go, and does the user path need streaming?
In: the document plus a 2k system prompt, 29,000 × $3 ÷ 1,000,000 ≈ $0.087. Out: 550 × $15 ÷ 1,000,000 ≈ $0.008. About $0.095 a document, $29 a day, $860 a month, 91% of it input. Latency: a few seconds of prefill plus 550 ÷ 50 = 11 s of decode, call it 14 s at p95. Nobody is waiting, so run it at upload in the background; streaming buys nothing here.
2. A 20-turn agent whose context grows by 1k a turn from a 4k start. Tokens in for the task, and what bounds it?
4k + 5k + … + 23k = 20 × (4k + 23k) ÷ 2 = 270k tokens in, against perhaps 4k out — about 68:1. A turn cap bounds the count; compaction of old turns bounds the growth; both are code in the loop, not lines in the prompt.
3. Product wants a one-paragraph justification returned with every label. Which of the four numbers move, and do you say yes?
Tokens out goes from 5 to about 80, 16×; decode time from 0.1 s to 1.6 s; output cost from nothing to 80 × $15 ÷ 1,000,000 ≈ $0.001 a call. Cost is not the objection. Say yes if someone reads the paragraph; if it is only logged, generate it on demand for the labels a person disputes and keep the hot path at 5 tokens.