The weekly review
Run an automation's 20-minute weekly review in a fixed order, and send each finding to the prompt, a gate, a rule or the connector.
On this page
The weekly review is 20 minutes in which an automation's owner reads four things in a fixed order and leaves with at most one change. Each of the four answers a different question, and each finding has one place to go: the prompt, a gate, a rule in the graph, the connector, or the log as "no change". A review without that order reads outputs until one looks wrong, then edits the prompt.
The hour that changed the prompt
Every Friday in the lead handler's first month the owner, an account manager, scrolled through the week's 90-odd runs for an hour. In week 3 she found two replies she disliked, one stiff and one that promised a site "live within two weeks", and edited the prompt on the spot: be warmer, and if in doubt about a date, hand the lead to a person.
The next week the person queue rose from 9 leads a week to 15, because the new clause made the model doubtful about any lead that named a deadline. Six extra hand-offs a week at 6 minutes is about 2.6 hours and $78 a month, and nobody linked the rise to the edit for three weeks, because the edit was written down nowhere.
Three faults, none of them carelessness:
- No order. An hour a week is 4.33 hours a month, $130, three times the 20-minute review's $43, spent on what is easiest to read, the sent replies, rather than on what changed.
- No record. The two replies were never saved as cases, so nothing could say whether the edit fixed them.
- No test. The edit went live without the eval set, so its side effect reached real leads first.
Twenty minutes in a fixed order
| Input | Minutes | What is read | The question |
|---|---|---|---|
| Sampled cases | 6 | the week's marked replies: 18 sampled at launch, the gate's rejects and edits from month 3; only the wrong ones opened | is the model wrong in a new way? |
| Escalations | 7 | the week's 9 or so, by class and reason, and the oldest still open | where does the procedure have a hole? |
| Spend against budget | 2 | month-to-date money and hours against the pro-rated line, and cost per lead | is it volume or behaviour? |
| Eval trend | 2 | the score of each change shipped and of the scheduled run; canary failures | did a change move the score? |
| Decision | 3 | at most one change, logged with its reason and who ships it | what changes this week? |
Sampled cases. In the lead handler's first two months, its launch setting, the reviewer reads one reply in five after it sends; the review reads her marks. What a sample can say depends on volume. At 18 replies a week and an assumed 2% wrong, it finds one wrong reply about every three weeks, and a clean week says little: zero errors in n reads is consistent with a rate up to about 3 ÷ n, so 0 of 18 allows anything to 17%. At this volume the sample yields cases to read; the rate comes from the eval set. From month 3 every reply is read before it sends, as inbound lead handling sets out, so all 92 a week carry a mark and the review reads the rejects and edits in the same 6 minutes. The online retailer's triage desk from module 8, a different company, checks about 250 routings a week at 4% mis-routed, about 10 errors, enough for a weekly rate by the sizing in sampling review and confidence thresholds.
Escalations. By class and reason, against the four-week average, with the age of the oldest open item, since every escalation has a clock. A reason seen three times in a week is a rule waiting to be written. A falling count is as much a finding as a rising one: leads that used to escalate are going somewhere else.
Spend against budget. Money and hours. The lead handler's tokens are about $0.50 a week, too small to watch in dollars, so watch cost per lead, $0.0054: a rising cost per lead is a change in behaviour, while rising spend at a flat cost per lead is growth. Hours: about 2.2 a week expected at launch and about 5.9 from month 3, when 20 hours a month of gate reads replace the sample's 4, from minutes the queue tool already timestamps.
The eval trend. The 60-case set from what "working" means runs on every change and weekly on a schedule, at about 60 × $0.0054 = $0.32 a run. The review reads one line per run: score, must-pass cases, version. A drop with nothing merged points at the provider. Canary failures from drift and silent failure are listed too: the alarm paged someone, and the review decides what changes.
What changes a prompt, and what changes a gate
| Finding | Goes to | Because |
|---|---|---|
| The same mistake twice, describable in one sentence | the prompt, with a new eval case | the model is wrong on inputs the procedure covers |
| The error rate, or the cost of one error, has moved with no single pattern | the gate or the sample rate | that is a cost decision about a rate |
| An escalation reason repeating | a rule in the graph | a branch should make any decision it can |
| A field empty, a format changed | the connector | the model was handed bad input |
A prompt change fixes a kind of mistake. Take the week-3 replies again, done properly. Suppose two replies in three weeks promised a delivery date, which the role's outputs forbid. That is a pattern, and the change is a hypothesis to test: save both leads as eval cases, edit the one instruction, run the set, and ship only if the score holds and both new cases pass. A gate would catch those replies before they went out, but it would not stop the model writing them, and each one would still cost the reviewer a rewrite.
A gate change accepts a rate of mistakes. When errors rise across many kinds, or one error starts to cost more, no sentence in the prompt addresses it; "be more careful" changes nothing a test can measure. The decision is module 4's inequality: gate every item when the error rate times the cost of an error exceeds the cost of reviewing one, and sample otherwise. Then find the cause, often upstream or a model update.
A rule takes a decision away from the model. Referral leads that keep escalating as "no CRM match" because the form sends the partner's address need a branch that matches on the referred company. It settles every one for nothing per lead and cannot drift; why such decisions belong in code is why reliability lives in the harness.
One change to the model step a week. If the prompt and the sample rate both change on Thursday, next Thursday cannot say which did what. A change needed sooner, to stop an incident, runs the eval set anyway and is logged at the next review.
In practice
Tell the team the slot, the order and the rule: Thursday at 16:00 for 20 minutes; sampled cases, escalations, spend, eval; at most one change, eval-run before it ships. Anyone who wants the prompt changed brings a case, not an impression, and nobody edits it in between except to stop an incident. Ask a vendor who runs it for the same four inputs weekly.
The mistake that costs money is editing the prompt on the strength of one output, between reviews, with no eval run and no log. The edit fixes the case in front of you, moves something else, and leaves nothing to trace the cost to.
Check yourself
1. Escalations doubled this week from 9 to 18, all "no CRM match", all from the partner-referral form. The owner proposes adding "search harder for the customer" to the prompt. Decide.
No prompt change: in the workflow the lookup is a graph step, and the model never searches. Nine extra hand-offs at 6 minutes is 54 minutes a week. Check what the referral form now sends in its email field, fix the mapping in the connector, and add a branch that matches on the referred company.
2. In month 2, the scheduled eval fell from 57 to 51 of 60 with nothing merged, and this week's sample found nothing wrong. Which do you believe, and what changes?
The eval. At about 95%, 60 cases have a standard error near 2.8 points, so a 10-point drop is over three standard errors, while 0 of 18 allows up to 17%. With nothing merged, suspect the model underneath: pin the dated version if offered and rerun. Until the score recovers, change the gate, not the prompt: read every reply before it sends, about 92 a week at 3 minutes, $138 a week.
3. Three findings this week: two replies promised dates, escalations are steady at 9, and cost per lead has doubled to $0.011 with volume flat. Which one change ships?
The prompt fix for the dates, eval-run with both leads added as cases, because a promised date can cost a client. The cost doubling is about $2.24 a month at 400 leads: a counter to read, not a change to ship. Find which input grew and bring the cause to next week.