Harnesses and multi-agent workflows · The harness

Five steps, one prompt

Multiply five measured step rates into a task success rate, then cost a check and one retry per step against the wrong reports it removes.

estimateeasyExercise · 3 min · 62 of 77

On this page

An agent that runs five steps from one prompt is as reliable as the product of its steps. The estimate turns five measured percentages into a daily count of wrong reports, then prices the harness from why reliability lives in the harness that removes most of them.

The situation

A finance team's expense agent runs one system prompt over five steps per report: extract amount, date and merchant from the receipt; map the expense to a policy category; check it against the policy limits; flag possible duplicates; write the reimbursement line. Each step is one model call of about 4,000 input and 200 output tokens. On a 200-receipt golden set the steps score 97%, 95%, 93%, 98% and 96%. The agent handles 3,000 reports a day, and a person spends about 5 minutes fixing each report it gets wrong.

A proposed harness adds a code check after every step and retries a failed step once with the check's error appended, about 300 extra input tokens. Assume the checks catch every failure and a retry fails independently of the first attempt.

Course constants: P_in = $3 and P_out = $15 per million tokens.

The question

Estimate the task success rate and the wrong reports per day with and without the harness, and the daily cost of the retries; then decide whether to build it.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Today the steps multiply: 0.97 × 0.95 × 0.93 × 0.98 × 0.96 ≈ 0.806. About 81% of reports are right, so roughly 580 a day are wrong: 48 hours of fixing.

With a check and one retry, a step fails only when both attempts do, so each failure rate squares: 0.0009, 0.0025, 0.0049, 0.0004, 0.0016. Multiplying the step successes gives about 0.990, or roughly 31 wrong reports a day, two and a half hours of fixing.

Retries run only on failed steps: 0.03 + 0.05 + 0.07 + 0.02 + 0.04 = 0.21 per report, 630 a day. Each reads about 4,300 tokens and writes 200: 2.7M input at $3/M is $8.10, and 126k output at $15/M is $1.90, about $10 a day. The base run is 60M input ($180) and 3M output ($45), $225 a day, so the retries add about 4%.

Decision: build it. $10 a day removes about 550 wrong reports and 45 hours of a person's time. The next prompt work goes to step 3: at 93% it carries nearly half of the remaining failures.

The estimate rests on both assumptions. It falls if a check misses failures (one that confirms a category exists but not that it fits catches only some) or if retries are not independent. Measure check coverage on the golden set, and read each step's retry success in the trace: near the step's own rate means independent; far below means a data problem a retry cannot fix.

Rubric

multiplies the five step rates into a task rate with the product shown, and converts it to wrong reports and hours a day
squares each step's failure rate for the retried case and multiplies the step successes
counts retries as the sum of failure rates, not one per step, and prices them with input and output separate
compares the retry cost with the base daily cost and the hours saved, and states a decision
names step 3 as the next target, with its share of the remaining failures
states both assumptions and the trace evidence that would lower the estimate

Where answers go wrong

"Every step is at least 93%, so the agent is above 90%." It averages where it should multiply; five steps that each look fine produce one wrong report in five.

"A retry on every step is 15,000 extra calls a day, which doubles the bill." The retry runs only when the check fails, 630 times a day, not 15,000. Pricing it as unconditional overstates the cost 24 times and rejects a harness that costs 4%.

"With retries the task is 99%, so ship it." The number is right and the reasoning is half done: it is stated as a fact rather than as the consequence of full check coverage and independent retries, with no plan to measure either. The reviewer is scoring whether the candidate knows what the 99% depends on.