Tokens and the context window
Count a request before sending it, split one window across system prompt, history, documents and output, and place the task where the model reads it best.
On this page
A model API accepts one sequence of tokens and returns another, and everything you send — the system prompt, the conversation so far, the documents you attach, the tool results, the space you leave for the reply — is flattened into that one sequence before the first token comes back. The sequence has a hard maximum, the window, and the window is a single budget that every part of the request draws on. The request that worked in testing with three documents and fails in production with forty is usually a budgeting failure, not a model failure.
What a token is
The model does not read characters or words. A tokenizer splits text into pieces from a fixed vocabulary: common words are one piece, rarer words two or three, and punctuation, whitespace and digits get pieces of their own. The course's conversion, fixed in the four numbers, is 1.3 tokens per English word — ¾ of a word per token, roughly four characters. A 500-word page is ~650 tokens; a 20,000-word document is ~26,000.
The ratio is worse wherever the vocabulary was not built. Code, with its brackets and identifiers, runs closer to one token per three characters. A JSON object with long keys is mostly keys. Numbers split digit by digit or in small groups, so a table of figures costs more than the prose around it. Text in scripts the tokenizer saw little of can cost two to three times the English rate. When a feature carries code, JSON or non-English text, measure rather than multiply.
Counting is cheap and exact. A tokenizer runs locally in microseconds per kilobyte, and the API's own count-tokens call returns the number the model will see, including the overhead of the message format and the tool definitions. Estimate with 1.3 to plan; count to send.
One budget, five claimants
The window is shared by the system prompt (role, rules, procedures), the tool definitions (every tool's schema and description, sent on every call), the history (every prior turn), the attached material (retrieved documents, tool results, files) and the output. The reply counts against the same window, so the tokens you want back have to be reserved before you send.
The request in the diagram sets itself a 64k budget — a number the engineer chose for this feature, below the window — and the sum of the bands is what the request costs: 49k in, 6k reserved out, 55k of the 64k. The free 9k is the margin that absorbs a longer-than-expected tool result. In a loop every band but the output grows per turn; memory and context management is the lesson on what to drop when it does.
Counting before sending
The mistake first. A summariser takes the documents a user attaches, concatenates them after a 4k system prompt, and asks for a 1k summary. In testing, three documents of ~8k tokens each — 28k in total — summarise cleanly. In production a user attaches forty. The request is 4k + 40 × 8k = 324k tokens: past the course's 200k window, from the four numbers, and rejected before the model sees it — or, worse, a client library truncates from the end, and the model summarises the first twenty-four documents while the reply reads as if it covered all forty.
The fix is a planner that runs before every call. Give each section a priority and a size; include the mandatory sections; pack the optional ones by priority until the budget minus the reserve is spent; refuse the request when the mandatory sections alone do not fit. The refusal is the point. A request that cannot fit should fail loudly in your code, not quietly in the model. The token budget planner exercise builds exactly this function, and the forty-document case becomes a decision — chunk, or retrieve the ten documents that matter — rather than an outage.
Why length costs twice
Attention, the mechanism in Vaswani et al., "Attention Is All You Need", lets every token read every earlier token. Processing the input — prefill — therefore grows faster than linearly with length, and the bill grows linearly with it: input tokens × P_in. The diagram's 49k of input costs 49k × P_in to read whether the model needed all of it or not, and it costs the same again on every turn of a loop, because the history is resent. Where the money goes does the cost sheet; the lesson here is that every token in the window is paid for, so the 12k of history nobody will reference again is not free.
Length costs time as well. Time to first token grows with the input; everything after it is paced by the decode speed, so a 1k-token reply takes 1,000 ÷ the decode speed — twenty seconds at the course's 50 tokens/s, from the four numbers — however short the input was. Latency budgets and streaming splits a budget into those two parts.
Lost in the middle
Fitting is not the same as being read. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", measured question answering over a set of retrieved documents while moving the one document that held the answer through the set. Accuracy was highest with that document at the start or the end of the context and fell by a double-digit margin when it sat in the middle — with every token still inside the window. The curve was a U, and it held across the main models tested.
The practice we draw from it. Instructions go first or last, never inside the material: a task statement placed after document nine of eighteen, buried inside 30k tokens of retrieved text, is in the worst seat in the window. State the task before the documents and restate it, in one or two sentences, after them. Order retrieved documents so the best-ranked sit at the edges rather than in rank order from the top, which puts rank 10 of 18 — often still relevant — exactly in the trough. And prefer fewer, better documents: ten that fit in 20k tokens beat forty in 80k, because the thirty extra are both paid for and badly read.
In an interview
What is being tested is whether you treat the window as a budget you allocate or as a limit you hit. A strong answer sizes the request out loud — "4k system, 3k tools, 30k retrieved, 6k reserved for output, 43k of a budget I set at 64k" — names what grows per turn, and says where the task sits and why. Cite Liu et al. by title when asked why placement matters; it is the result the interviewer expects you to know.
The mistake that loses points is "it fits in the window" with no count. It ignores the reserve the reply needs, the history that grows per turn, and the finding that a document in the middle is read worse than one at the edge. The second is treating the token count as a word count: a table of figures or a JSON blob costs two or three times the prose rate, and an estimate that uses 1.3 everywhere is wrong exactly where it matters.
Check yourself
1. A support assistant carries a 6k system prompt and 2k of tool definitions, reads three 4k tool results per turn, and writes ~500 tokens per turn. You cap its requests at 100k. At which turn does the planner start dropping tool results?
Per turn: 3 × 4k + 500 ≈ 12.5k. Fixed: 8k, plus 2k reserved for the reply, leaves 90k for history. Seven turns are 87.5k and fit; turn 8's tool results take the history past 90k, so the planner starts dropping old tool results, or compaction starts, at turn 8 — long before anyone thinks of the window as the limit.
2. A legal reviewer wants one question answered over 18 retrieved clauses of 1.5k tokens each. Rank order puts the best clause first and the tenth-best in the middle. Where do you put the question and the clauses?
18 × 1.5k = 27k tokens: it fits. Put the question first, then the clauses with ranks 1–5 at the front and ranks 6–10 at the end — 11–18 in the middle, or dropped — and restate the question in one sentence after the last clause. The clause the answer needs is then at an edge; a clause in the trough is one you already ranked as least likely to matter.
3. A feature is sized at 1.3 tokens per word, but the documents are spreadsheets exported as CSV. Do you trust the estimate?
No. Digits and separators tokenise at one token per two characters or fewer, against about four for prose, so a CSV runs two to three times the prose estimate. Count three real exports with the tokenizer before sizing the budget: a 20k-word estimate can be 60k tokens of CSV, and that changes whether the request fits at all.