Testing a harness without a model
Test a harness with a scripted stub, replayed transcripts and property checks on the trace, and keep the real model out of the test suite.
On this page
A test that calls a real model is slow, spends tokens on every run, and does not give the same answer twice, so when it fails nobody knows whether the code changed or the sample did. The harness is the part that has to be right every time, and none of it needs a model to test: the model is a parameter, and a test passes a different one. Three substitutes cover what a harness does, and each answers a different question.
Why the test suite never calls the model
Take 40 harness tests, each a six-turn run with an average context of 20,000 tokens. One suite run reads 40 × 6 × 20,000 = 4.8 million input tokens at P_in, and a team that pushes 30 times a day reads 144 million a day to test code, not the model. Time is worse. At the course's latency figures from the four numbers, about a second to first token and 50 tokens/s, a turn with 200 output tokens takes about 5 s, a test 30 s, and the suite 20 minutes run serially.
Flakiness is worst. Output varies between runs even at temperature 0, for the reasons sampling and determinism gives. If each test passes 98% of the time against a real model, the 40-test suite is green 0.98⁴⁰ ≈ 45% of the time. A suite that is red half the time with no code change teaches everyone to rerun it until it is green, and from then on it tests nothing.
The same 40 tests against stubs read no tokens, finish in well under a second and give one answer. The runner behind this course's code exercises works that way: it imports each solution, calls solve() on every case in TESTS and fails the build on one mismatch, and no case touches a network.
The split this enforces is clean. Harness tests check the code with the model held fixed; evals check the model and the prompt with the harness held fixed. A test that varies both cannot say which one broke.
The scripted stub
class Stub:
"""The model interface, answered from a fixed list of replies."""
def __init__(self, replies):
self.replies, self.sent = list(replies), []
def __call__(self, messages):
self.sent.append(list(messages)) # what the harness sent, for assertions
if not self.replies:
raise AssertionError("script ran out: the harness made a call the test did not expect")
return self.replies.pop(0)
A stub gives two things. The first is control of every reply, so the cases a real model produces once a month are one line each: an unknown tool, a block cut off at the output cap, the same call ten times, a two-call cycle, a tool that raises, a reply that crosses the token budget. Every TESTS list in this module is a stub test of that kind; the tool-dispatch loop is the smallest.
The second is a record of what the harness sent. A test can assert on the request, not only the outcome: the error result for block 2 went back on the next send, the context after compaction is under budget, the timeout result names the tool that timed out.
The mistake is a stub whose first reply is a final answer. It passes and exercises nothing. Each stub test should drive the harness down the path it is named after and assert on the trace field that proves it got there: the status, the log entry, the attempt count. A stub that runs out of replies raises, and should, because the harness made one more call than the test planned for, which is a harness bug or a wrong test, and both need a person.
Replayed transcripts
A stub tests the paths someone thought of; a recorded transcript tests the paths production took. Record the requests and replies of a sample of real runs, from staging or from redacted traces, and replay them: the replayer answers each request with the reply recorded for the same request, looked up by a hash of the request in canonical form, with messages, tool definitions and parameters serialised with sorted keys.
The lookup must be by hash, not by position. If a refactor changes what the harness sends, by reordering tool definitions, rewording an error result or compacting a turn earlier, the hash misses and the replay fails on that turn and names it. A positional replayer hands back the recorded reply anyway and passes while the behaviour changed underneath it. A changed request is also a changed cache prefix, so the miss is worth reading even when the change was intended; prompt caching prices what one costs.
When the change was intended, re-record, but only after the eval suite has run on the new behaviour. Re-recording first turns whatever the new version does, regressions included, into the baseline.
Property checks on the trace
Stubs and transcripts check cases one at a time. Property checks cover cases nobody wrote. Generate scripts from a seeded random generator, mixing calls to real and unknown tools, malformed and unclosed blocks, raising tools, repeated calls and final answers at random turns; run the harness on each; and assert what must hold on every run:
- the run ends within
max_turns, with exactly one of the five statuses - every call in the log has one result, in call order
- no request exceeds the window minus the output reserve
- every turn is either sent verbatim or folded into a summary that is sent, so compaction loses nothing silently
- no DAG step starts before its dependencies, and a resumed run never reruns a recorded step
- the same seed gives the same trace
A thousand generated runs of about ten turns each is ten thousand turns of plain Python, a few seconds. The seed is the reproduction: when a property fails at seed 4,817, shrink the script by deleting replies while it still fails, and add the short version as a fixed stub case. Properties find the combinations a person does not script, such as the unknown tool on the last permitted turn, or a malformed block in the same reply as the call that crosses the budget. Run the determinism property in two processes and it also catches the scheduler that walks a set of step names, whose order changes with each process's string hashing.
In an interview
The interviewer is testing whether you can make agent code testable and say what each kind of test can prove. The answer in one line: "the model is injected, so stubs cover the paths, replayed transcripts catch regressions in what we send, property checks hold the invariants, and evals measure the model." Then say why the model stays out: tokens on every push, minutes per suite, and a pass rate that is a probability.
The mistake that loses points is "we test the agent by running it on a few prompts against the model." That is an eval with no labels and a flaky test at the same time. The second is mocking the HTTP layer, which tests the client library's serialisation and breaks on every upgrade; stub at the model interface the harness already has, as a harness in 200 lines defines it.
Check yourself
1. A CI suite has 60 harness tests that call the real model, each passing 98% of the time. How often is it green, and what do you change?
0.98⁶⁰ ≈ 0.30, so seven runs in ten are red with no code change. Move all 60 onto stubs and replays, which pass or fail for one reason, and run the real model in the nightly eval suite, where its variance is measured rather than suffered.
2. A refactor reorders the tool definitions, and all 50 replayed transcripts fail on their first request. Re-record or revert?
Revert, unless the reorder was the point. Every request's bytes changed, so every cached prefix in production misses once and the model reads a different prompt. If it was intended, run the evals on the new order first and re-record only if they hold.
3. Which property catches a compaction planner that drops the previous summary, and why does a stub test of one compaction miss it?
The conservation property: every turn is sent verbatim or folded into a summary that is sent. A run that compacts once has no earlier summary to drop, so the stub passes; a generated run long enough to compact twice breaks the property on the second fold, which is the bug budgeted compaction is built to avoid.