Memory and context management
Work out the turn at which an agent fills its window, then choose context editing, compaction, a scratchpad or external memory by what each costs and loses.
On this page
An agent loop spends the context window the way a process spends memory: every turn allocates and nothing frees. A window shared by the system prompt, the tool definitions, every earlier call, every tool result and the output still to come is full after a predictable number of turns, and the number is small. Context management is the set of mechanisms that free space, and each one costs something different: a model call, lost detail, a cache miss, or a design decision made up front.
The arithmetic: full around turn 55
Take an agent whose tool results average 4 KB. At roughly four characters per token, as tokens and the context window set out, that is about 1,000 tokens per result. It makes three calls a turn, so 3,000 tokens of results, and writes about 500 tokens of output per turn across the calls themselves and any reasoning around them. Each turn therefore adds 3,500 tokens to the history. The system prompt and tool definitions are 6,000 tokens, resent every turn.
The context on turn n, before the model answers, is
6,000 + 3,500 × (n − 1)
The window is the course's 200k tokens, from the four numbers, and the model needs room to answer, so reserve 8,000 for output. The last turn that fits is the largest n for which 6,000 + 3,500(n − 1) + 8,000 is within the window, which is n = 54, and on turn 55 the send is 195,000 tokens with 5,000 left for an answer that may need more. Call it full at turn 55.
Look at where the tokens are. Tool results are 159,000 tokens, four-fifths of the window, and the model finished using most of them within a turn of receiving them. The model's own calls and decisions are 26,500. The thing the model needs to remember, what it did and what it learned, is a small fraction of the thing it is carrying.
Every token in the window is read again on every turn, so across 54 turns this agent reads about 5.3 million input tokens at P_in (54 × 6,000 + 3,500 × 54 × 53 ÷ 2 = 5,332,500); freeing the window is the same act as cutting the bill.
Four mechanisms, in the order they fire
Context editing: drop old tool results
The cheapest mechanism replaces a tool result, once it is a few turns old, with a stub: [result of find_free_slots, turn 12, 1,000 tokens, dropped]. The call stays, so the model knows it asked; the data goes, because the model already acted on it. No model call is needed and nothing is summarised.
Keeping the last ten turns' results and stubbing the rest, the window at turn 55 is 6,000 + 30,000 + 27,000 + 8,000 reserved = 71,000 tokens, with 129,000 free, and it stops growing by 3,500 a turn; it grows by the 500 tokens of the model's own turn.
What it loses: the model cannot re-read a dropped result. If the task needs the eight free slots from turn 2 on turn 40, the model calls the tool again, one more round trip. Edit results, never the model's own turns.
Compaction: summarise old turns
When the model's own turns are the bulk, or when the dropped results held something the model still needs, compaction rewrites turns 1 to k into one summary message: what was asked, what was tried, what was learned, what remains. It is a model call with turns 1 to k as input and a summary as output, so it costs k × 3,500 tokens at P_in plus the summary at P_out, and it loses whatever the summary left out.
Compaction at turn 30 of the agent above, summarising turns 1 to 20 into 2,000 tokens, brings the next send from 111,000 to 43,000 and buys about nineteen more turns before the window fills; the compaction at turn 30 exercise prices it, and budgeted compaction implements it.
The summary is a prompt the model writes to itself, and it is wrong in the ways summaries are wrong: it drops the exact value, the file path, the error text. Mitigate by telling the summariser what to keep, verbatim, in a list: identifiers, numbers, open questions.
A scratchpad: a file the model writes
Both mechanisms above decide for the model what it will remember. A scratchpad lets the model decide: give it write_notes(text) and read_notes(), tell it to record findings and plans there, and the notes survive editing and compaction because they live outside the window. The model re-reads them when it needs them, at the cost of one call. The failure mode is a model that writes nothing, or everything; the note-taking instruction needs the same care as a tool description.
It fires by design, before the first turn, on any task that will run for more than a few dozen turns.
External memory: a store with retrieval
Beyond one run, memory is a store: a database of facts, past runs, or documents, queried by key or by similarity, and read into context on demand. This is retrieval, and the model does not know your data treats it as the search problem it is. For an automation that runs for days across many invocations the founder-facing course covers the state design in memory and state for long-running automations.
What each mechanism does to the cache
Prompt caching matches a prefix, and every mechanism here changes bytes early in the conversation. Context editing that stubs a result on turn 12 makes everything after turn 12 a cache miss on the next send, written again at C_w × P_in instead of read at C_r × P_in. Compaction rewrites the whole history, so the next send is one cache write of the new, shorter prefix.
The practical rule is to edit in batches: stub ten turns' worth of old results at once, every ten turns, not one turn's results every turn, so the cache is lost ten times in a hundred turns rather than a hundred. The prompt caching lesson has the arithmetic for what one miss costs at a given prefix size; at 100k tokens it is not small.
Which mechanism, when
| Pressure | Mechanism | Fires |
|---|---|---|
| Results are the bulk, task is short-lived | context editing | at a threshold, in batches |
| The model's own turns are the bulk, or dropped results are still needed | compaction | at a threshold, rarely |
| The task runs for dozens of turns | scratchpad | from turn 1, by design |
| Knowledge spans runs | external memory | by query, as retrieval |
Editing and compaction show up in the trace as a drop in input tokens per turn, which is how you confirm they fired.
In an interview
The interviewer is testing whether you can say when the window fills and what you do about it, with a number. Do the turn-55 arithmetic out loud, point at the 159,000 tokens of stale results, and name editing before compaction because it needs no model call. Then say what each loses and what each does to the cache.
The mistake that loses points is reaching for a bigger window. A window twice the size fills around turn 110, and because cost is quadratic in turns the run reads about four times the input on the way there, 21.6 million tokens against 5.3 million; the task did not need the stale results at any size. The second mistake is compacting every turn, which pays a summarisation call and a cache write on every round trip.
Check yourself
1. Tool results average 12 KB instead of 4 KB, one call per turn, 500 tokens of output. On which turn does the window fill with an 8,000-token reserve?
12 KB is about 3,000 tokens, so each turn adds 3,500 again: 6,000 + 3,500(n − 1) + 8,000 within the window gives n = 54, the same turn. The window does not care how the 3,500 is split, but the fix differs: one large result per turn means editing a single result frees 3,000 tokens, so edit aggressively, keep the last three rather than the last ten.
2. The agent runs 40 turns, and context editing is applied every turn to the result from ten turns earlier. Roughly how many cache misses does that cause, and what would you change?
Thirty: every turn from 11 onwards changes a byte at turn n − 10, so the prefix beyond that point misses every time. Batch it: stub results every ten turns in one edit, for three misses in forty turns, and keep the window within a few tens of thousands of the same size.
3. A research agent will take about 120 turns and must cite exact figures from pages read on turn 5 in its final answer on turn 120. Which mechanisms do you set up before turn 1?
A scratchpad with an instruction to record every figure it may cite, with its source, as soon as it reads it; context editing of results older than ten turns, in batches; and compaction only if the model's own turns push past the threshold. The figures survive in the notes, which the model re-reads on turn 119, and the window never comes close to full.