Workflows, state and the harness · Graphs

Which steps get a model

Sort a nine-step invoice SOP into model steps and code steps, then estimate the weekly rework of that design against an agent that does all nine.

casemediumExercise · 3 min · 23 of 75

On this page

An SOP can read like nine judgments and turn out to be seven rules. Which steps get a model decides how many invoices come back each week, and deterministic first is the lesson this practises.

The situation

The Friday invoice run bills 40 projects a week from the time tracker's export. By hand it takes the operations manager about 6 minutes an invoice, $120 a week at $30 an hour. A wrong invoice costs about $50 of rework. The SOP has nine steps:

Export last week's time entries.
Group hours by client and project.
Apply each client's rate card.
Flag hours that look wrong: more than 10 in a day, or 50% above last week.
Turn each project's free-text notes ("fixed nav bug, call w/ Sam re Q3") into two or three lines a client can read.
Add tax by the client's country.
Decide whether the invoice needs a cover note: over the purchase-order cap, or 20% above last month.
Write the cover note, about 4 a week.
Create the invoice in the accounting system and email it.

A contractor proposes one agent with six tools that does all nine. Assume a model step is right 95% of the time, a tested code step every time, and a code check after a model step catches 80% of its errors.

The question

Decide which steps get a model and which never do, and estimate the weekly rework under your design and under the contractor's agent.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Two steps get a model: 5, because the notes are free text and the output is prose a client reads, and 8, because a cover note is a judgment about tone. The other seven are code. Steps 1, 2, 3 and 6 are lookups and arithmetic. Steps 4 and 7 sound like judgment, but the SOP states them as rules, so code applies them exactly. Step 9 is a side effect, so code does it once, with a key.

Each model step gets a check. After step 5, code confirms every line maps to a billed project and contains no figure absent from the entries. Step 8's note goes to the operations manager before it is sent, 4 a week at about 2 minutes, so its errors cost minutes, not rework.

Rework in this design: step 5 lets through 5% × 20% = 1% of invoices, 0.4 a week, $20 a week, about $87 a month.

The agent picks the order and performs every step through tools, so all nine are model steps with no independent check: 0.95⁹ ≈ 0.63, and 37% of invoices carry an error. That is about 15 a week, $740, more than six times the $120 a week the run costs by hand. Granting that the tools make the export and the send reliable leaves seven: 0.95⁷ ≈ 0.70, about 12 a week, $600.

Build the graph with two model steps. If the time notes were already client-ready, step 5 would become code too and the run would need no model at all, only a script.

Rubric

Assigns all nine steps to model or code with a reason each, and keeps the side effect in step 9 in code.
Recognises steps 4 and 7 as written rules, not judgments.
Puts a check or a person after each model step and says what the check looks for.
Computes the weekly rework for both designs from the stated rates, with the arithmetic shown.
Compares the agent's rework with the $120 a week the task costs by hand.
Names the evidence that would move a step from model to code.

Where answers go wrong

"Step 7 needs judgment about the client relationship, so give it to the model." The SOP states the rule: over the cap, or 20% above last month. A model applying it adds a miss rate to a decision code makes exactly, and a missed cover note on an over-cap invoice is the $50 of rework the run exists to avoid.

"Rate cards have exceptions, so the model should apply step 3." Exceptions are data. A rate card with per-client overrides is a table lookup, and hours times rate is arithmetic a model gets right most of the time and code gets right every time.

"The agent is fine if its prompt tells it to double-check each invoice." The check runs in the same context, by the component that made the error, so it is not independent. 0.95⁹ stays where it was, and so does the $740.