Tools and the agent loop · Loops and context

Compaction at turn 30

Estimate how many turns a compaction at turn 30 buys an agent before its window fills again, and what it costs against the tokens it saves.

estimatemediumExercise · 3 min · 19 of 77

On this page

Compaction is a model call that rewrites the past. It costs tokens now to save tokens later, and whether it pays depends on how long the run goes on.

The situation

A research agent makes three tool calls a turn, each returning about 4 KB, and writes about 500 tokens of output per turn, so the history grows by 3,500 tokens a turn. Its system prompt and tool definitions are 6,000 tokens. The harness reserves 8,000 tokens of the window for the model's answer. At turn 30 the harness compacts: it sends turns 1 to 20 to the model with a 500-token instruction and receives a 2,000-token summary, which replaces those twenty turns; turns 21 to 30 stay verbatim. Runs of this agent typically go to about turn 60.

Course constants, restated: about four characters per token, so 4 KB is about 1,000 tokens; a 200k-token window; P_in = $3 per million input tokens and P_out = $15 per million output tokens; a cache read at C_r = 0.1 × P_in and a cache write at C_w = 1.25 × P_in. Ignore caching for the main estimate and then say what it changes.

The question

How many more turns does the compaction buy before the window is full again, what does the compaction call cost, and how many input tokens does it save over turns 31 to 54 compared with not compacting?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Without compaction the context on turn n is 6,000 + 3,500(n − 1). With the 8,000 reserve the last turn that fits is n = 54 (6,000 + 185,500 + 8,000 = 199,500), as memory and context management works out.

After compaction at turn 30, the context on turn 31 is 6,000 + 2,000 (summary) + 10 × 3,500 (turns 21 to 30) = 43,000, growing by 3,500 a turn: 43,000 + 3,500(n − 31) + 8,000 ≤ 200,000 gives n − 31 ≤ 42.6, so turn 73 is the last that fits. The compaction buys 73 − 54 = 19 turns, enough to reach the typical turn 60.

The compaction call reads 20 × 3,500 + 500 = 70,500 tokens and writes 2,000: 70,500 × $3/M ≈ $0.21 plus 2,000 × $15/M = $0.03, about $0.24.

Savings over turns 31 to 54. Without compaction the inputs sum to 24 × 6,000 + 3,500 × (30 + 31 + … + 53) = 144,000 + 3,500 × 996 = 3,630,000 tokens. With it, 24 × 43,000 + 3,500 × (0 + 1 + … + 23) = 1,032,000 + 966,000 = 1,998,000. The saving is about 1.63 million input tokens, $4.90 at P_in, twenty times the compaction's cost.

Caching changes the ratio, not the decision. With the history a cache hit each turn at C_r, the saving falls to about $0.50, and the compaction forces one cache write of the new 43,000-token prefix, about $0.16 at C_w. Still positive. The evidence that reverses it: runs that end before turn 50, which fit without compaction; cached, the saving of about $0.02 a turn needs twenty turns to repay the $0.24 + $0.16.

Rubric

derives the turn at which the uncompacted window fills, with the reserve included
computes the post-compaction context and the new fill turn, and states the turns gained
prices the compaction call with input and output separated
sums the input tokens over turns 31 to 54 with and without compaction, arithmetic shown
says what caching does to the saving and to the compaction's own cost
names the run length below which compaction does not pay

Where answers go wrong

"Compaction saves 70,000 tokens, the size of the turns it removed." It saves that much on every later turn, not once; the sum over 24 turns is 1.6 million. The candidate forgot that the history is resent each turn.
"Compact every ten turns to keep the window small." Each compaction costs a model call and a cache write of the whole prefix; at every ten turns that is six calls and six cache misses in a 60-turn run for a window that was never close to full. Compaction fires at a threshold, not on a timer.
A correct token saving with no mention of the turns gained. The run was going to fail after turn 54 and typically needs 60; the first value of compaction here is that the task finishes at all, and the $4.90 is second.