Sampling review and confidence thresholds

Whatever clears a gate without a person still needs someone looking at it, and two numbers decide how much: the share of finished items a reviewer reads afterwards, and the score above which an item never waits for a person. Both are usually set by feel, "check about one in ten" and "auto-send above 0.9", and both have arithmetic that ends in reviewer hours a day. The example is the online retailer's desk from approval gates, a different company from the agency: 3,000 tickets a day, staff time assumed at $30 an hour loaded.

A sample measures; it does not filter

"We review 10% of what the router does, so we catch its mistakes." It catches 10% of them. The other 90% reach customers exactly as if nobody were looking. A sample does one job: it tells you the error rate, and so tells you when the rate changes. If errors must be stopped, the item needs a gate, and the test from the last lesson decides which:

error rate × cost if wrong  ≥  review cost per item   →  gate every item
error rate × cost if wrong  <  review cost per item   →  sample

The ticket router files all 3,000 tickets into 12 categories. Assume a mis-route costs 3 minutes of an agent's time to re-route, $1.50; the labelled set puts the mis-route rate at 4%; checking one routed ticket takes 30 seconds, $0.25. Expected loss is 4% × $1.50 = $0.06 a ticket, a quarter of the review cost. Sample.

How big a sample: 1 in N

Read a sample in errors, not percentages. One error where you expected two says nothing; twenty where you expected ten says the rate has doubled. Assume a review window needs about 10 errors at the normal rate before its rate can be told apart from double. Then:

reviewed per window    n = 10 ÷ error rate
sampling rate          1 in N,  N = items per window ÷ n

At 4%, n = 10 ÷ 0.04 = 250 routed tickets per window. The window length is the decision, and error cost makes it:

The shorter window buys time. If a change doubles the mis-route rate, the extra mistakes cost 3,000 × 4% × $1.50 = $180 a day, and a weekly read finds the doubling about three days after a daily one would: $540 an incident. Assume one such incident a month, from a prompt edit, a model update or a renamed upstream field. The daily window costs $53.50 a day more, about $1,600 a month, to save $540. The router is sampled 1 in 84, 18 reviewer minutes a day.

Two consequences surprise people. A lower error rate needs a larger sample: at 1% the window needs 1,000 tickets, because it still needs ten errors and they are rarer. And incidents follow changes, so review should too: for the day after any prompt, model or connector change, read at the daily rate, 250 tickets and $62.50 once, then drop back.

A confidence cut, calibrated on 200 cases

The desk drafts a reply for every ticket with a score attached, and the tempting rule is "send anything above 0.9", on the theory that 0.9 is right nine times in ten. That is a guess about a number the model produced; structured outputs says what a self-reported confidence is worth until checked. So check it. The support lead labels 200 recent drafts right or wrong at about a minute each, 3.3 hours, grouped by score:

This model happens to be roughly calibrated, but that does not place the cut. The cut comes from the same two costs as a gate. Assume a wrong reply that goes out costs $15: a reopened ticket, 10 minutes of a person, and a $10 goodwill credit. A review costs 2 minutes, $1. A band belongs above the cut when its wrong rate is under $1 ÷ $15 = 6.7%. The 2.5% and 5% bands qualify; the 15% band does not. The cut is 0.90 because of $15 and $1, not because 0.9 sounds safe. At $45 a wrong reply the break-even is $1 ÷ $45 = 2.2%, and no band on this table qualifies.

Precision, recall and reviewer hours at the cut

Expand

At a 0.90 cut, 60 of 200 labelled drafts are sent and 58 of those are right; 140 go to a person, 97 of them right and 43 caught wrong. At 3,000 tickets a day that is 900 sent with about 30 wrong at $15 each, and 2,100 reviewed at 2 minutes, 70 reviewer hours100%−+FitSVG (opens in a new tab)CloseEsc

Pinch to zoom · drag to panScroll to zoom · drag or arrow keys to pan · 0 fits · Esc closes

Precision is the share of what is sent that is right: 58 ÷ 60 = 96.7%. Recall is the share of right drafts that go without a person: 58 ÷ 155 = 37.4%. Above the cut each error costs $15; below it every draft costs 2 minutes, right or wrong. At 3,000 a day, 30% are sent, 900, with about 30 wrong, $450 a day; the other 2,100 take 70 reviewer hours. Those hours are not new: the same people answered those tickets before, from a blank page.

Moving the cut trades one cost for the other:

Dropping to 0.80 saves 20 reviewer hours, $600, and sends 90 more wrong replies, $1,350. Rising to 0.95 spends $300 more on review to save $225 of errors. On these costs 0.90 is cheapest, by $75 and $750 a day.

Two cautions. A band of 20 drafts with one error could show none or three on the next 20, so label another 100 drafts scoring 0.85 to 0.95, where the line is drawn, before trusting it. And scores move whenever the prompt or model does, so the 200-case check reruns on every change, like the eval set. The 900 replies above the cut are then sampled, at a rate worked from their own error cost by the method above.

In practice

To the team: "Above 0.90 the reply sends. On 200 labelled cases that is 96.7% precision and 37% recall: 900 sent a day, about 30 wrong at $15, and 70 review hours for the rest. A cut at 0.80 would cost $750 a day more." To the reviewer: the sample is a measurement, so log every error with its category, because the count is the product.

The mistake that costs money is a cut chosen by feel and never priced. The same 0.9 is right for $15 replies and wrong for $45 refunds, and nobody can tell which until the confusion table is filled in. The second is treating a 10% sample as protection: 90% of the errors still reach the customer.

Check yourself

1. The desk drops the $10 goodwill credit, so a wrong reply costs $5. Where does the cut go, and what does a day cost?

The break-even wrong rate rises to $1 ÷ $5 = 20%. The 15% band now qualifies and the 25% band does not, so the cut drops to 0.80: 1,500 sent with 120 wrong at $5, $600, plus 50 reviewer hours, $1,500, for $2,100 a day. Both neighbours cost $2,250, 0.90 in review and 0.70 in errors.

2. A better prompt drops the router's mis-route rate from 4% to 1%. Weekly or monthly reads?

The window needs 10 ÷ 0.01 = 1,000 tickets either way: 143 a day weekly, $36 a day; 33 a day monthly, $8. A doubling now costs only 3,000 × 1% × $1.50 = $45 a day, and a monthly read finds it about 12 days later, $540 an incident, against $840 a month saved. Monthly at one incident a month; weekly at two or more.

3. The agency's lead handler in how to use this course launches with an account manager reading one sent reply in five, at 3 minutes. Assume 2% of drafts are wrong, and a wrong reply loses the lead one time in four, a $1,500 first project. Is one in twenty safe?

Neither rate is right, because this is a gate. A wrong reply costs $1,500 ÷ 4 = $375 on average, so 2% × $375 = $7.50 of expected loss a reply, against $1.50 to read it first. A read after sending counts the 8 wrong replies a month and stops none of them, about $3,000 of expected loss. This is the analysis the agency runs in month 3, after a wrong reply costs it a deal: from then on every reply is read before it sends, 400 × 3 minutes, 20 hours, $600 a month in place of the $120 sample. With the owner's 20-minute weekly review, $43, the handler then costs about $785 a month instead of $305, the saving falls from $1,695 to about $1,215, and the build still pays back in about a month.