Designing for "the model might be wrong"
Apply the human-AI interaction guidelines to a drafting feature, set the measured error rate below which it may act alone, and track edits before send.
On this page
At 92% on the golden set, the drafting feature is wrong on 8 drafts in 100: at one tenant's 48,000 drafts a day, 3,840 wrong drafts in front of agents every day. No release brings that to zero, so the question is what happens on those 3,840: who notices, at what cost, and whether it reaches a customer. Amershi and colleagues' "Guidelines for Human-AI Interaction" (CHI 2019) sets out 18 guidelines; seven of them, applied as four design moves, carry most of the weight for a feature that drafts and a person sends.
The mistake: a disclaimer and an auto-send
The usual first proposal: a line on every draft reading "AI-generated, may contain mistakes", and the simplest tickets sent without review.
The disclaimer changes no draft. The half-point factual rate, one golden-set draft in 200, is 240 wrong facts a day with or without it, and a warning printed on everything is read on nothing within a week. The contract clause is about the errors, not the labels.
The auto-send is worse, and its arithmetic is the trap. Sending unreviewed saves the whole 2-minute reply; a wrong reply that reaches a customer costs a 20-minute escalation (jobs to be done). 2 ÷ 20 = 10%, so on time alone auto-send pays below 10% wrong: per 100 tickets at today's 8%, 200 agent-minutes saved and 8 × 20 = 160 lost. That prices every wrong reply the same, but a tone miss costs an awkward sentence and a wrong price costs the contract clause. The decision turns on the costliest class, and the agent is the only part of the system that catches it.
Show what it did
Guideline 11 is "make clear why the system did what it did", and guideline 2 "make clear how well the system can do what it can do". For a draft, both come down to sources. The draft lists the help articles it drew on, and each price, date and policy statement links to the sentence it came from, so the agent checks a refund window with a glance rather than a search through 200 macros. A fact with no source is marked for checking.
Citations are one of retrieval's advantages in RAG against fine-tune; here they make each of the 240 wrong facts a day findable in seconds. "87% confident" never does that: it does not say which sentence to check.
Make correction one action
Guideline 9 is "support efficient correction", guideline 15 "encourage granular feedback". Three corrections, each one action:
- Edit inline. Once inserted, the draft is ordinary text. No edit mode.
- Regenerate. One key, with the previous draft one step back.
- Report a fact. Select the wrong span, one click; the report carries the draft, its sources and the correction.
On a 2-minute cycle a ten-second, four-step report flow gets skipped, and the golden set loses the field failures that evals as the feature's test suite says it should grow from. If one click gets even 1 in 10 of the 240 daily factual errors reported, that is 24 corrected cases a day, about 170 a week, all in the class that blocks launch and where a 200-case set is thinnest.
Make undo cheap
Guideline 8 is "support efficient dismissal", guideline 16 "convey the consequences of user actions". The draft never overwrites the agent's text, and dismissing it is one key. Sending is the one-way door, so the reply is held 10 s with Undo: 10 s on a reply the customer waited minutes for, nothing for the agent, who has moved on, and a number, undos per 1,000 sends, that counts wrong replies caught at the last moment.
Never auto-act above a measured error rate
Guideline 10 is "scope services when in doubt". As a product rule: the feature acts alone only on a slice where the costliest failure class is measured below its threshold. Here the class is facts and the threshold is the eval's blocking rate, 0.5%. Two words do the work.
Measured. A rate of 0.5% cannot be read off 200 examples. One error in 200, today's count, is already 0.5%: enough to pass the launch gate with an agent reading every draft, not to remove the agent. Zero in 200 still allows a true rate up to about 1.5%: zero failures in n examples bounds the rate below about 3 ÷ n at 95% confidence, the rule of three. Showing under 0.5% with zero failures takes 600 examples; estimating it within ±0.3 points takes 2,000, by the evals lesson's margin formula, 2 × √(0.005 × 0.995 ÷ 2,000) ≈ 0.003.
Slice. The FDE's "simplest 30% of tickets" is 14,400 drafts a day here, and the golden set has no number for it. Shadow mode produces one: for two weeks each slice draft is marked "would auto-send" while the agent works as usual, and the log records what they did. A draft whose numbers, dates or prices the agent changed counts as a factual error. Agents miss some, so the support lead audits 200 unchanged sends for the miss rate, about 200 minutes at an assumed minute a draft, roughly $100 at $30 an hour.
Even at the line, 0.5% of 14,400 is 72 wrong facts a day reaching customers, so the tenant signs off the threshold.
The edit-before-send metric
Two numbers on every sent reply: whether the draft went out unchanged, and if not, the edit distance as a share of its characters. The first is the draft acceptance input of the North Star, 0.40.
Alone, it can be gamed by drafting only easy tickets, as that lesson shows. And 40% unchanged against 92% acceptable is not a 60% error rate: agents add a name or soften an opening line on drafts the support lead would sign. Assume a week reads 40% unchanged, 35% light edits (under half the characters) and 25% heavy (more than half rewritten). The heavy quarter, 12,000 drafts a day, cost reading time and saved nothing; cohorts and funnels traced the same loss to long drafts.
One sub-count matters most: edits that change a number, date or price. It is the field estimate of the factual error rate, measured on every send rather than 200 examples, and the label shadow mode runs on. Beside it sits the guardrail: reply time must not rise above 2 minutes.
In an interview
"Our model is wrong 8% of the time; how do you design for it?" tests whether you turn a rate into a daily count and a design into actions. Say: "8% of 48,000 is 3,840 wrong drafts a day, 240 on facts. Drafts cite sources. Edit, regenerate and report are one action each, and reports become eval cases. Sends are held 10 s for undo. Nothing auto-sends until a slice measures under 0.5% factual on at least 600 examples."
The mistake that loses points is a disclaimer, which moves liability and not error, or auto-send above a confidence score nobody has checked against outcomes. The second is an acceptance metric with no fact-edit count, which rises when agents stop checking.
Check yourself
1. The FDE reports zero factual errors in 250 shadow-mode password-reset drafts, checked by the support lead, and asks to auto-send that slice from Monday. Decide.
Not yet. Zero in 250 bounds the rate below 3 ÷ 250 = 1.2%, more than twice the 0.5% line. Run shadow mode until 600 checked drafts show no factual error: 350 more checks, about six hours of the lead's time at an assumed minute a draft. Then auto-send with the 10 s hold, and stop the moment the slice's undo rate or fact-edit count crosses 0.5%.
2. The tenant's operations lead objects that the 10 s send hold slows every reply. Estimate its cost and value, and decide.
Cost: 10 s on a reply the customer waited minutes for, and no agent time. Value: if 0.5% of 48,000 sends are undone, 240 wrong replies a day never reach a customer, 240 × 20 minutes = 80 agent-hours of escalation avoided. Keep it; if the measured undo rate is under 0.05%, 24 a day, shorten it to 5 s.
3. The team proposes "raise drafts sent unchanged from 40% to 50%" as next quarter's goal. Decide what the goal should say instead.
Unchanged share rises if the team drafts fewer, easier tickets, so pair it: heavy rewrites from 25% to 15% of drafts, with intent coverage held and the fact-edit count flat or falling. That moves 4,800 drafts a day from saving nothing to saving time, and it cannot be met by drafting less.