Implement the agent loop so an unknown or failing tool becomes a result the model reads, and a turn cap ends the run with a status and the partial log.
codemediumExercise · 3 min · 69 of 77
On this page
The loop is twenty lines, and most production agents that fail badly fail in those twenty lines. A scripted stub stands in for the model, so every way the loop can go wrong is a test case rather than an incident.
A team's agent loop crashed in production when the model called a renamed tool, and separately ran for 400 turns on a task with no exit. The rename was get_order to fetch_order in Tuesday's deploy; an example in the system prompt still used the old name, and for two hours 6% of runs ended in a KeyError with nothing in the trace. The 400-turn run is the one the loop that ran for 400 turns takes apart; successful runs take 3 to 7 turns.
The loop wanted can do neither: unknown tools go back to the model as error results, and a turn cap ends the run with a status the trace can read. The team wants it proved before it ships, with no model in the test: the stub is a list of replies the loop plays in order, and each tool is a plain callable.
def solve(script, tools, max_turns):
"""script: stub replies in order, ("call", name, args) or ("final", text).
tools: name -> callable(**args). Returns (status, answer, log), where
status is "done" or "budget" and log is [(name, result), ...]."""
A tool that raises is logged the same way, as its exception type and message:
def lookup(order_id):
raise TimeoutError("orders service timed out after 5 s")
solve([("call", "lookup", {"order_id": "A-17"}), ("final", "orders is down")], dict(TOOLS, lookup=lookup), 5)
-> ("done", "orders is down", [("lookup", "error: TimeoutError: orders service timed out after 5 s")])
A strong answer · Rubric · Where answers go wrong · Solution
One for over range(max_turns); a final returns ("done", text, log); a call looks the tool up, and a missing name or a raised exception becomes a string result in the log — the loop never raises. Falling out of the loop returns ("budget", None, log). Twenty lines.
The cap is the loop's bound, not a check inside it, so no path runs a third call when max_turns is 2, and a final on the last permitted turn is still done. Dispatch sits in a helper: an unknown name returns error: unknown tool nope; a known one runs inside try, and except Exception turns any failure into error: TimeoutError: …. The type name is for the model, which should retry a timeout and fix its arguments after a TypeError. Catching Exception rather than everything leaves an interrupt free to stop the process.
The log is appended in call order, one entry per call, and survives the budget exit, because a capped run's trace has to say what it did. The stub ignores the conversation; a real loop appends each call and result to the messages before the next send, and that one line is all that changes when a client replaces the stub, as a harness in 200 lines shows.
What the loop never does: retry itself, sleep, or keep state outside the function. A module-level log would carry one test's entries into the next, and in production one run's calls into another's trace. The exits beyond the turn cap belong to agent loops and stop conditions.
while True with break on final: passes the first two tests and fails the third, calling past the cap of two until the script runs out and the loop crashes. Against a real model it never returns. That loop is why the exercise exists.
A try around the whole loop that returns an error status when anything raises. The renamed tool now ends the run quietly instead of loudly, the partial log is discarded, and the model never sees the error result it would have recovered from on the next turn.
Checking the turn count after dispatching rather than bounding the loop by it. The third test then logs three calls for a cap of two, and every capped run in production spends one turn more than its budget allows.
Python 3 · standard library33 lines · 7 of 7 tests passing
"""Tool-dispatch loop against a stub model — failures become results, and a turn cap ends the run."""
def lookup(order_id):
"""Fixture: a tool that fails the way a real one does when its backend is down."""
raise TimeoutError("orders service timed out after 5 s")
def dispatch(tools, name, args):
"""Run one call. Every failure becomes a result string the model can read; nothing escapes."""
if name not in tools:
return f"error: unknown tool {name}"
try:
return tools[name](**args)
except Exception as exc: # a tool's failure is data for the model, not a crash for the loop
return f"error: {type(exc).__name__}: {exc}"
def solve(script, tools, max_turns):
"""script: stub replies in order, ("call", name, args) or ("final", text).
tools: name -> callable(**args). Returns (status, answer, log), where
status is "done" or "budget" and log is [(name, result), ...]."""
log = [] # invariant: one entry per call made, in call order
for turn in range(max_turns): # the cap is the loop bound, not a check inside a while
reply = script[turn] # the stub ignores the history; a real model would read it
if reply[0] == "final":
return ("done", reply[1], log)
_, name, args = reply
log.append((name, dispatch(tools, name, args)))
return ("budget", None, log) # the partial log survives for the trace
TOOLS = {"add": lambda a, b: a + b, "upper": lambda s: s.upper()}
The cases that ran
TESTS = [
(([("call", "add", {"a": 2, "b": 3}), ("final", "5")], TOOLS, 5),
("done", "5", [("add", 5)])),
(([("call", "nope", {}), ("final", "gave up")], TOOLS, 5),
("done", "gave up", [("nope", "error: unknown tool nope")])),
(([("call", "upper", {"s": "a"})] * 3, TOOLS, 2),
("budget", None, [("upper", "A"), ("upper", "A")])),
# a tool that raises is caught and logged like an unknown one, and the run goes on
(([("call", "lookup", {"order_id": "A-17"}), ("final", "orders is down")], dict(TOOLS, lookup=lookup), 5),
("done", "orders is down", [("lookup", "error: TimeoutError: orders service timed out after 5 s")])),
# a final on the last permitted turn is still done, not budget
(([("call", "add", {"a": 1, "b": 1}), ("final", "2")], TOOLS, 2),
("done", "2", [("add", 2)])),
# the log keeps call order across tools
(([("call", "upper", {"s": "ok"}), ("call", "add", {"a": 4, "b": 5}), ("final", "OK 9")], TOOLS, 5),
("done", "OK 9", [("upper", "OK"), ("add", 9)])),
# a zero-turn budget sends nothing
(([("final", "never sent")], TOOLS, 0),
("budget", None, [])),
]
