Workflow scripts and deterministic harnesses
Write a harness as a DAG of typed steps with a budget each, a resume file and a stop point, and say what a failed run re-pays when it restarts.
On this page
A workflow script is the harness written down: a DAG of steps, each a model call or plain code, with typed inputs and outputs, a budget per step, a resume file, and stop points where a person decides. The script owns the order, the retries and the state; the model owns one judgment inside one step. The shape pays on the bad day: when step 5 of 7 fails, the rerun starts at step 5 with steps 1 to 4 read from disk, not paid for again.
The mistake: one long agent run
A team turns each month's product changelog into a blog post. Version one is a single agent with a long system prompt: read the changelog, pick what matters, outline, draft, check every claim against the changelog, publish. It works on most runs. On the run where the fact-check call times out, the whole task fails and the rerun starts from nothing. It re-reads the changelog, samples a slightly different selection, writes a different outline and a different draft, and the claims the first run had half-checked no longer exist. Nothing survives, because the only place it lived was the context of a process that died.
There is a quieter failure too. The fact-check reads prose and returns prose, so nothing downstream can tell a passed check from a model that wrote "looks good".
A script is a DAG of typed steps
Each step is a record the scheduler can read:
@dataclass
class Step:
name: str
kind: str # "model" | "code" | "person"
needs: list[str] # upstream steps: the edges of the DAG
run: Callable # input -> output; a model call or plain code
output_schema: dict # JSON Schema the output must satisfy
budget: Budget # max tokens in and out, max turns
retries: int = 1 # attempts after a failed check
needs makes the edges, and the scheduler runs steps in topological order, the ordering in cycles and topological order. Steps with no path between them can run in parallel, and a cycle in needs is rejected before anything runs. output_schema makes the output typed: the harness validates it before any later step reads it, as structured outputs and JSON Schema describes, so a malformed draft fails at the step that wrote it, not three steps later. budget applies the turn and token exits of the agent loop to one step.
run is an interface. The scheduler calls it with an input and gets an output or an error; whether a model, a regex or a person implements it is invisible to the scheduler. That is designing to an interface applied to a harness, and it pays three ways: a model step becomes code when its rule turns out to be deterministic, a test swaps in a stub that returns scripted outputs, and a step moves to a cheaper model without the scheduler changing.
Seven steps, costed
The post pipeline as a script:
| Step | Kind | In | Out | Weighted |
|---|---|---|---|---|
| 1 Fetch changelog | code | 0 | 0 | 0 |
| 2 Select entries | model | 6,000 | 300 | 7,500 |
| 3 Outline | model | 3,000 | 500 | 5,500 |
| 4 Draft | model | 4,000 | 2,000 | 14,000 |
| 5 Fact-check | model | 9,000 | 600 | 12,000 |
| 6 Editor approves | person | 0 | 0 | 0 |
| 7 Publish | code | 0 | 0 | 0 |
| Total | 22,000 | 3,400 | 39,000 |
"Weighted" is input plus five times output, the ratio of P_out to P_in assumed in the four numbers, so one post costs 0.039 × P_in. Steps 4 and 5 are two-thirds of it. Step 5 reads the changelog entries straight from step 1, not step 2's selection, because a fact-check that sees only what the model selected cannot catch a claim the selection got wrong; that edge is what makes the pipeline a DAG rather than a chain. Each model step's budget is about twice its typical use: the draft is capped at 4,000 output tokens, 80 seconds of decode at the course's 50 tokens/s, also from the four numbers, and a step that hits its cap fails like any failed check.
The resume file
When a step's output passes its check, the harness writes it to run.json under the step's name with a hash of the step's inputs. A rerun walks the DAG in the same order. For each step it hashes the current inputs and compares: equal means skip and read the output back; different means run the step, which changes its output and so the input hash of everything downstream.
Step 5's call returns overloaded errors until the retry policy gives up, and the run stops with steps 1 to 4 in run.json. The rerun pays for step 5 only: 12,000 weighted tokens against 39,000 for starting over, a 69% saving. The larger gain is that it checks the same draft the editor will read, not a new one sampled from a new outline.
The failure decides the resume point, not the step number. A transient failure — a timeout, an overloaded API, a crashed process — resumes at the step that failed. A check failure whose cause is upstream resumes upstream: if step 5 reports a claim the changelog does not contain, the defect is in step 4's draft, so the rerun is step 4 with the finding attached, then step 5 again. The hashes do the rest: the new draft changes step 5's inputs, while the changelog and the outline are read back untouched. The tiny DAG scheduler exercise computes a resume point from a list of failures.
Stop points and side effects
Step 6 is a person. The harness reaches it, writes the state and exits; no process waits, no context is held, and the editor can take two days. Approval is a write to run.json from a command or a button, and the next run resumes at step 7 because step 6 now has an output. A stop point is the cheapest place for a judgment that should not belong to the model; approval gates places them in a business process.
Step 7 publishes, which is a side effect. If the process dies after the CMS accepted the post and before run.json recorded it, the rerun publishes again, unless the call carries an idempotency key derived from the run id and the step name, so the CMS returns the post it already has. Every side-effecting step gets one; idempotency is the general case.
Script, graph tool or framework
Visual graph tools and state-machine libraries offer the same parts: nodes, typed state, edges, checkpoints, an interrupt for a person. Which of them should carry a given SOP is taught in a graph tool versus a state-machine library for one SOP. The choice changes the skin, not the properties above, and a team that can name the properties can evaluate any of them. A plain script with all of them is about 200 lines; a harness in 200 lines writes it.
In an interview
Asked to design a pipeline, draw the DAG first and put a number on each node: kind, tokens in and out, budget. Then say where it resumes: "each checked output goes to a resume file under a hash of its inputs, so a failure at step 5 re-pays step 5 — 12k of 39k weighted tokens — and a defect found at 5 but made at 4 reruns from 4". Then the person and the side effect: the stop point, and the key on publish.
The mistake that loses points is a pipeline with no persisted state between steps. Every late failure becomes a full rerun, outputs that were already checked are re-sampled, and a stop point for a person is impossible because nothing survives the wait.
Check yourself
1. A nine-step pipeline's steps cost 2k, 5k, 5k, 8k, 30k, 4k, 4k, 2k and 1k weighted tokens, and step 8 fails transiently on 3% of runs. Is a resume file worth building for that alone?
Without one, a step 8 failure re-pays all nine steps, 61k; with one, 2k. Per 1,000 runs: 30 failures × 59k saved ≈ 1.8M weighted tokens, about 3% of the 61M those runs cost. Marginal on cost alone; the stronger reason is that a full rerun re-samples step 5, the 30k step every later step was checked against.
2. The editor rejects the draft at step 6: "the second section misstates the pricing change". Where does the run resume?
At step 4, with the note added to its input. The new hash invalidates 4, and the new draft invalidates 5 and 6; steps 1 to 3 are read back. Cost: 14k + 12k = 26k weighted. Resuming at 5 would re-check the same wrong draft. The miss also belongs in step 5's golden set, since the fact-check passed it.
3. A teammate proposes merging outline and draft into one call to save a round trip. Do you?
Count what the merge loses. It saves one call and step 3's 3k of input, under a tenth of the pipeline. It loses the outline as a checked, persisted output: a draft failure now re-pays the outline, and there is no point to check structure — every selected entry covered — before 2,000 tokens of prose are written. Keep them separate unless the outline check never fires.