Why reliability lives in the harness
Multiply step success rates into a task success rate, and move retries, gates, budgets, ordering and idempotency out of the prompt into code you can test.
On this page
On the five-era ladder, this module is the last rung, harness workflows: code owns the loop and the model owns the judgment inside one step. A model step that is right 95% of the time sounds dependable until five of them are chained, and then the task is right 77% of the time. A sentence in the prompt barely moves that number, because the prompt is read by the component that is failing. A check and one retry in code around each step moves it to about 99%, which is why retries, gates, budgets, ordering and idempotency belong in the harness, and why the harness is what we test.
Five steps at 95% is a 77% task
Take an accounts-payable agent that turns an invoice into a ledger entry in five steps: extract the line items, match each line to a purchase order, assign a ledger code, flag anomalies, and write the approval note a person reads. Version one is the loop from agent loops and stop conditions: one system prompt lists the steps, and the model reasons, acts and observes until it writes the note — the loop of Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models (opens in a new tab)". The loop is fine. The arithmetic is not.
Measure each step on a golden set, as evals as the unit test of LLM work describes, and suppose each is right 95% of the time. The task is right only when all five are, and when errors are independent the probabilities multiply:
after step 1 0.95 95.0%
after step 2 0.95² = 0.9025 90.3%
after step 3 0.95³ ≈ 0.857 85.7%
after step 4 0.95⁴ ≈ 0.815 81.5%
after step 5 0.95⁵ ≈ 0.774 77.4%
At 1,000 invoices a day that is about 226 wrong ledger entries, and no single step looks broken.
The mistake: fix it in the prompt
The first fix everyone tries is a sentence: "Before moving to the next step, check your work carefully." It cannot help much: the check is done by the model that made the error, in the same context, and the errors that matter are the ones it cannot see. Nothing enforces the sentence, no log line says when it was skipped, and there is no function to test.
Suppose a week of prompt work lifts every step from 95% to 97%. The task goes to 0.97⁵ ≈ 85.9%: still one invoice in seven wrong. Chaining multiplies, so per-step gains of a few points cannot outrun the exponent.
A check and one retry, in code
Version two keeps the five prompts and moves control into a harness. After every step, code checks the output before the next step reads it: the lines validate against the schema and sum to the invoice total; every purchase order exists; the ledger code is in the chart of accounts; every figure in the note appears in an earlier step's output.
When a check fails, the harness retries the step once with the check's error appended — "line totals sum to 1,180; the invoice total is 1,280". If the retry fails too, the run stops and the invoice goes to a person with the failed check attached. If the check catches every failure and the retry fails independently of the first attempt, a step fails only when both attempts do:
step failure = 0.05 × 0.05 = 0.0025 → step success 99.75%
task success = 0.9975⁵ ≈ 0.988 → about 99%
From 226 bad entries a day to about 12. The retry runs only on the 5% of steps that fail their check, so a task averages 5.25 calls instead of 5: five percent more tokens for 21 points of task success.
What the arithmetic assumes
The 99% rests on two assumptions, and each is engineering work.
The check sees the failure. If the ledger-code check confirms that a code exists but not that it is the right one, it might catch 70% of that step's failures; then step success is 0.95 + 0.05 × 0.7 × 0.95 ≈ 98.3%, and five checks that weak leave the task at 0.983⁵ ≈ 92%. The harness is exactly as reliable as its checks, which is why reading the response spends a lesson on them.
The retry is independent. The same bytes sent to the same model often fail the same way, for the reasons in sampling and determinism; appending the check's error makes the second attempt a different request. A failure caused by the input — the purchase order is not in the system — fails every retry, and needs a person, not a third attempt. A step whose retries usually fail, in the trace, has a data problem.
Retries, gates, budgets, ordering and idempotency
Each concern has a prompt-line version that holds most of the time and a code version that holds every time:
| Concern | As a prompt line | In the harness |
|---|---|---|
| Retry | "if a step fails, try again" | one retry on a failed check, error attached |
| Gate | "make sure the totals add up" | the check between steps; a second failure stops the run |
| Budget | "be efficient" | a token and turn cap per step |
| Ordering | "match the PO before coding" | the edges of the DAG |
| Idempotency | "never post an invoice twice" | a key on the posting step |
The last row is one a prompt cannot do even in principle: a posting call that timed out may have succeeded, and a retry posts twice unless the tool takes a key — the chain from retries, timeouts and rate limits and idempotency.
Anthropic's "Building effective agents (opens in a new tab)" draws the same line as workflows versus agents: a workflow orchestrates models and tools "through predefined code paths", while an agent directs its own process. A harness is the workflow side applied step by step, and its prompts get shorter: the instructions about order, retrying and checking have left them.
The harness is what you test
A prompt-only agent has one test: the whole task on the golden set, a pass rate near 77%, and no indication of which step failed. A harness has two kinds. The model steps get evals, one golden set per step, which is where the 95% came from. The harness gets unit tests against a stub model that returns scripted outputs: a bad extraction triggers exactly one retry, a second stops the run, a replayed post writes one ledger row. They run in milliseconds with no API key; testing a harness without a model builds them. Which rung of the automation ladder a task deserves, and how to run the result as a virtual employee, is taught in rung 4 of the automation ladder.
In an interview
"How would you make this agent reliable?" tests whether you multiply before you answer. Say it with the numbers: "five steps at 95% is 0.95⁵, about 77% per task; a code check and one retry per step makes each step 99.75% and the task about 99%, for 5% more calls". Then name the two assumptions — the check has to see the failure, and the retry has to be independent — and what you do when the second breaks: route to a person.
The mistake that loses points is answering with the prompt: "add 'double-check your work'", "add examples", "use a bigger model". Each nudges the per-step number and leaves the exponent alone.
Check yourself
1. Ten steps, each right 98% of the time, no checks. Product wants 99% per task. A better model, or a harness?
0.98¹⁰ ≈ 81.7%. Without checks, 99% per task needs 99.9% per step, a twentyfold cut in error that no model swap delivers. A check plus one retry gives 1 − 0.02² = 99.96% per step and about 99.6% per task, at 10.2 calls instead of 10. Harness.
2. Step 3 is right 95% of the time, and its check catches half its failures. You have a week: lift the prompt to 97%, or the check to 90%?
Step failure is the missed failures plus the caught ones whose retry also fails. Today: 2.5% + 0.125% ≈ 2.6%. Prompt at 97%: 1.5% + 0.045% ≈ 1.5%. Check at 90%: 0.5% + 0.225% ≈ 0.7%. The check wins by a factor of two, and it is code you can unit-test.
3. Step 2's retries succeed only 30% of the time, against about 95% for every other step. Raise step 2 to three attempts?
No. A retry that fails 70% of the time is not facing a random error; the same input produces the same failure. If the purchase orders are missing or the lookup returns the wrong supplier, fix the data or the tool, and route the step to a person after one failed retry. Three attempts would add a third call to every one of those failures and fix none.