Harnesses and multi-agent workflows · The harness

Resume after step 5

Decide where a crashed seven-step statement harness resumes when the step that failed is not the step that broke, and cost the rerun against the deadline.

casehardExercise · 3 min · 63 of 77

On this page

A resume file makes "start from where it failed" cheap, as workflow scripts and deterministic harnesses shows. The case is harder when the step that failed is not the step that broke.

The situation

A monthly statement harness runs seven steps for 2,000 business customers: (1) pull 120,000 transactions, code, 40 minutes; (2) categorise them, model, 1,200 calls of 5,000 tokens in and 1,000 out, 3 hours; (3) compute totals, code, seconds; (4) draft each customer's narrative, model, 3,000 in and 400 out per customer, 2 hours; (5) fact-check each narrative against the totals, 4,000 in and 100 out per customer, 2.5 hours; (6) the finance lead reviews, 4 hours; (7) email statements.

Step 5 crashed at customer 1,412 when the API kept returning overloaded errors past the retry budget. Of the 1,411 checked, 37 failed: the narrative's totals disagree with step 3's. All 37 have refunds from one card network that step 2 labelled as purchases, and a fix is ready. The resume file holds steps 1 to 4 and step 5's results so far, each under a hash of its inputs. Statements are due in 26 hours.

Course constants: P_in = $3 and P_out = $15 per million tokens.

The question

Decide where the run resumes, which stored outputs you keep, and what the rerun costs in money and hours against the deadline.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Two failures, two resume points. The crash is transient and alone would resume at customer 1,412. The 37 mismatches are a step 2 defect: step 5 noticed it, step 2 made it. Resuming at 1,412 would check 589 more narratives built on wrong categories and send the 37 known-bad ones.

Rerun step 2 in full with the fix: 6M input and 1.2M output, $18 + $18 = $36, 3 hours. Not only that card network: the defect's scope is inferred from 37 cases, and $36 is cheap insurance. Step 1's pull is on disk and stays.

Then the hashes decide. Step 3 recomputes in seconds, and a customer whose categories did not change keeps its step 4 hash, its narrative and any step 5 result. Assume about 200 customers change. Step 4 for them: 600k in, 80k out, about $3 and 12 minutes. Step 5 for those 200 plus the ~530 unchanged customers never checked: 730 × 4,000 ≈ 2.9M in, 73k out, about $10 and under an hour. Total about $49 and 4 hours, then 4 hours of review: done in under 9 of the 26.

A full rerun from step 2, $93 and 7.5 hours, also fits. The targeted path wins on review: the finance lead reads a diff of 200 statements with the reason attached, not 2,000. Step 7 sends with a key per customer and month, so a crash mid-send emails nobody twice.

What reverses it: step 5 failing on customers without that network's refunds. The defect is then wider than step 2, so stop and rerun steps 4 and 5 for everyone.

Rubric

separates the transient crash from the upstream defect and gives each its resume point
rejects resuming at customer 1,412 alone, and says what it would ship
chooses the scope of the step 2 rerun and justifies it against its cost
uses input hashes to limit downstream reruns, and says which stored outputs are kept
costs the rerun in money and hours, review included, against the 26-hour deadline
keys the send step and names the evidence that would widen the rerun

Where answers go wrong

"Resume at customer 1,412; that is what the resume file is for." It treats the step that failed as the step that broke. The 37 known-wrong statements go out, along with about 15 more among the 589 unchecked customers at the same rate.

"Rerun everything from step 1 to be safe." It repeats a 40-minute pull that was never wrong and re-drafts 2,000 narratives the finance lead must re-read in full, turning an 8-hour fix into a 12-hour one with less margin for review.

"Fix the 37 narratives by hand and resume." It patches what the fact-check happened to see. Step 3's totals are wrong for every customer with that network's refunds, including those whose narrative did not mention the refund and so passed the check.