A harness in 200 lines
Assemble the parser, dispatcher, stop checks, compaction planner, retry policy and DAG scheduler into one harness, and say which part owns each failure.
On this page
A harness is the code around the model. It parses what the model wrote, runs the tools, decides when to stop, keeps the context inside its budget, retries what failed and puts the steps in order. Each of those jobs is a pure function of under forty lines, and this module's five exercises are five of them; with the retry policy from module 3 and a trace, they make one program of about 200 lines in which the model is a parameter and every failure has one part that owns it.
The version everyone writes first
The first harness is usually the naive loop from agent loops and stop conditions with three things bolted on: a try around the body, a retry inside the loop, and the API client built inside the function. At the first incident the trace says "error" and nothing else: a malformed call, a tool timeout, a rate limit and a stuck loop all landed in one except clause, and the inner retry has doubled the turn cap.
The fix is to split the loop where failures differ, so that each kind has one function that detects it and one status that reports it.
Two loops, one inside the other
The outer structure is a DAG of steps, the shape workflow scripts and deterministic harnesses draws: each step is code, a model call or a person, with a typed input and output, a budget and its own retry limit. The inner structure is the agent loop inside each model step: send, parse, dispatch, check, repeat.
The split is what makes the reliability arithmetic in why reliability lives in the harness work. A step that succeeds 95% of the time gives a five-step chain 0.95⁵ ≈ 77%. One retry takes the step to 1 − 0.05² = 99.75% and the chain to 0.9975⁵ ≈ 98.8%, provided the two attempts fail independently. That retry lives in the scheduler, which is the only part that sees a step fail as a whole.
The model is a parameter
Nothing in the harness imports a vendor client. The model is a function from messages to a reply carrying text and token counts, passed in as an argument: an adapter over a real API in production, a list of scripted replies in a test. That is designing to an interface applied to the one dependency that is slow, priced per token and not deterministic, and it is what lets testing a harness without a model test every part below with no network.
def run_step(model, tools, history, limits):
keys, spent = deque(maxlen=limits.window), 0
for turn in range(1, limits.max_turns + 1):
reply = send_with_retries(model, history, limits.retry) # 408, 429 and 5xx only
spent += reply.tokens_in + reply.tokens_out
entries = parse(reply.text) # calls and error entries
if not entries:
return Result("done", reply.text, turn, spent)
if spent > limits.token_budget:
return Result("token_cap", None, turn, spent)
results = [dispatch(tools, e) for e in entries] # never raises
keys.append(turn_key(entries, results))
if repeating(keys):
return Result("no_progress", None, turn, spent)
history = history + [assistant(reply.text), tool_results(results)]
history = compact_if_due(history, limits, model) # a plan, then one call
return Result("turn_cap", None, limits.max_turns, spent)
One turn, five parts
The parser runs on every reply. The JSON tool-call parser returns calls and error entries in one list, so a malformed block is answered with a result naming it, and a block cut off by the output cap is reported, not guessed at.
The dispatcher turns every entry into a result string. Unknown names and raised exceptions, as in the tool-dispatch loop, and parser error entries become results the model reads next turn; the loop never raises.
The stop checks are the three exits: the turn cap as the loop's bound, the token cap against the step's budget, and the no-progress detector, which keys each turn on call and result. done is checked before the token cap: if the model answered on the call that crossed the budget, the tokens are already spent, and discarding the answer wastes them.
The compaction planner runs only when the history passes a trigger. With the numbers from memory and context management, 3,500 tokens of history a turn and a trigger at 105,000, it fires on turn 30. Budgeted compaction plans the cut: the newest 10 turns stay, everything older goes to the summariser, and the summary gets what a 37,000-token budget leaves, 2,000 tokens. The next compaction is (105,000 − 37,000) ÷ 3,500 ≈ 19.4, so twenty turns later. When the planner refuses because the kept turns alone overflow, the harness stubs old tool results, which needs no model call, and plans again.
Retries live in two places, and neither is the loop
Transport failures are retried inside send_with_retries, on the schedule from the retry-with-backoff policy: 408, 429 and 5xx with doubling waits, never a 400. The step sees them only as latency.
Step failures are retried by the scheduler. A tiny DAG scheduler runs the steps in one topological order, gives each its own attempts and stops at the first step that uses them up. A model step that ends in turn_cap or no_progress is rerun with a fresh history, not continued, because continuing a stuck context is how a turn cap of 25 becomes 50.
A step with side effects gets no retry without an idempotency key. publish has a limit of 0 for that reason until it carries a key checked before the post goes out, as idempotency describes.
The trace is the output
Every step returns a Result with the status first. The scheduler appends one record per attempt and, when a step succeeds, writes its output to the resume file under a hash of its inputs, so a rerun skips exactly the steps whose inputs have not changed. Per-step tokens priced at P_in and P_out are the cost sheet; the status counts are the dashboard that observability and tracing builds.
Where the 200 lines go
| Part | Built in | Lines |
|---|---|---|
| Parser | JSON tool-call parser | ~35 |
| Loop, dispatcher and caps | Tool-dispatch loop against a stub model | ~45 |
| No-progress check | No-progress detector | ~15 |
| Compaction planner | Budgeted compaction | ~15 |
| Transport retries | Retry-with-backoff policy | ~20 |
| Scheduler | A tiny DAG scheduler | ~35 |
| Trace, resume file, glue | this lesson | ~35 |
The model adapter, tools and prompts are not in the 200 lines; they change per feature, and the harness does not.
In an interview
The interviewer is testing whether you can write the loop as code with the model as one function in it. Say it in one breath: "the parser turns text into calls and errors, the dispatcher turns calls into results and never raises, three stop checks end the loop with a status, compaction keeps the window inside a budget, and the scheduler orders, retries and records where to resume." Then say where retries live: transport in the client wrapper, steps in the scheduler, never in the loop or the prompt.
The mistake that loses points is answering "we use a framework" without being able to say what its loop does on an unknown tool, a malformed call or a turn cap. The second is a retry inside the loop, which doubles every cap and hides the failure from the trace.
Check yourself
1. A model step succeeds 90% of the time and its failures are independent. How many retries per step does a six-step pipeline need to finish 99% of runs?
One retry makes a step 1 − 0.1² = 0.99 and the run 0.99⁶ ≈ 0.94, short. Two make a step 0.999 and the run 0.999⁶ ≈ 0.994. Two retries, then check independence in the trace: if the failing 10% are inputs the step can never handle, every retry fails the same way and the fix is upstream.
2. With the compaction numbers above, how many compactions does a 100-turn run make, and what does each read?
Four, on turns 30, 50, 70 and 90. The first reads turns 1 to 20, 70,000 tokens; each later one reads the 2,000-token summary plus twenty turns, 72,000. Each also writes about 2,000 tokens at
P_outand forces a cache write of the new prefix atC_w, so compacting a little on every turn after 30 would pay both about seventy times instead of four.
3. publish times out after the post has gone live, and the scheduler marks the step failed. Retry it, resume from it, or neither?
Neither, as written: both post twice. Give the step an idempotency key from the run id and step name and have it check for the key before posting. Then resuming from
publishis safe.