Tools and the agent loop · Loops and context

The loop that ran for 400 turns

Find what is wrong with an agent loop that polled a deployment for nearly 400 turns, cost 1,600 times a normal run, and ended only when the context filled.

critiqueeasyExercise · 3 min · 18 of 77

On this page

A deployment agent ran for nearly 400 turns on a task that normally takes six. Nobody noticed until the invoice did.

The situation

A team's release agent has eight tools, including trigger_deploy(service) and get_deploy_status(deploy_id), and a system prompt and tool set of 3,000 tokens. A normal run is six turns: trigger, poll twice, read the log, post a summary, stop. Last Thursday the deploy backend hung and get_deploy_status returned the same 2 KB status object, {"state": "pending", …}, for an hour. The agent called it again, and again, with the same argument, each call and its result adding about 500 tokens to the history. The run ended on turn 395, when the send filled the window and the API rejected it. The loop's code is:

while True:
    reply = model(messages)
    if reply.stop_reason == "end_turn":
        return reply.text
    messages += run_tools(reply.calls)

The trace for the run records the final answer (none) and the total tokens. Assume a 200k-token window, about four characters per token, P_in = $3 per million tokens and P_out = $15 per million; a normal run reads about 25,000 input tokens.

The question

Say what is wrong with this loop and its trace, and what the team ships instead, with the cost of Thursday's run worked out.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Turn n sends 3,000 + 500(n − 1) tokens, which fills the window on turn 395. The 394 sends before it read 394 × 3,000 + 500 × 394 × 393 / 2 = 1,182,000 + 38,710,500 ≈ 39.9 million input tokens, about $120 at P_in, against $0.075 for a normal run: 1,600 times the usual. Output is about 400 short calls, under $0.50.

The loop has one exit and the model owns it. Three are missing, in the order they would have fired. A no-progress check, the same call returning the same result twice in a row, ends this run on turn 3 for about $0.03; it compares results, since a normal run also polls twice. A turn cap at twice the p95 of successful runs, say 15, ends it at turn 15 for about $0.29. A token budget per task, say 100,000 input tokens, ends it on turn 16. "Context full" is not a stop condition; it is the absence of one.

The second fault is the trace. It records the answer and the total, not which exit fired, so a stuck loop looks like a long task. The first field of every run should be the terminal state: done, no_progress, turn_cap, token_cap or error; a weekly count per state is the dashboard.

The third fault is design: polling is a job for code, not a model. get_deploy_status should block up to ten minutes before returning pending, so the model is called only for a decision. With that and the three exits from agent loops and stop conditions, Thursday's run ends on turn 3, twenty minutes in, as no_progress, paging an engineer.

Rubric

prices the run with the quadratic sum shown and compares it to a normal run
names the three missing exits and says which would have fired first and at what turn
identifies "context full" as the absence of a stop condition rather than one
adds the terminal state as the first field of the trace and says what it is aggregated into
moves the polling into the tool so the model is called only when there is a decision

Where answers go wrong

"Add a max_turns of 400 so it cannot run forever." It already stopped near 400; the cap has to come from the distribution of successful runs, which is six, so a cap of 400 changes nothing about Thursday and only converts a window error into a cap error.
"The model should have noticed it was repeating itself." The model did notice, each turn, that the deploy was pending and polled again, which is what its instructions implied. The repetition is for code to detect, with a hash of the last call and its result, because the check has to hold the 400th time.
A correct set of exits with no change to the trace. Next Thursday the loop stops at turn 15 and the team still cannot say why, so the cap gets raised "to let it finish". The field that says which exit fired is what makes the cap tunable.