Human in the loop · Gates and review

A threshold set at 0.9 by feel

Find why an auto-approval cut chosen because 90% sounds safe loses money on refunds, and decide what replaces it, with the daily cost of each option.

critiqueeasyExercise · 3 min · 31 of 75

On this page

A confidence cut that sounds safe is a claim about two costs nobody wrote down. This one costs more each day than having no automation at the step at all.

The situation

For one month, before anyone measured its error rate, the online retailer this module uses in place of the agency auto-approved refund requests under $100 when the model's eligibility score was 0.9 or more; the founder picked 0.9 because "90% sounds safe". About 200 such requests arrive a day. Requests scoring below 0.9 went to a reviewer, who decides one in 2 minutes at a loaded $30 an hour. An ineligible refund that is approved costs its amount, about $21 on average: the desk's 240 refunds a day average $45, but the 36 over $100, which go to the gate, average $180. Nobody sampled the auto-approved refunds. After a month, the support lead labelled 200 past requests:

Score	Requests	Eligible	Not eligible
0.98 and up	60	59	1
0.90 to 0.98	90	79	11
below 0.90	50	30	20

Assume, as the course does, that a review window needs about 10 errors at the normal rate before a change in that rate shows.

The question

Find the defects in how the 0.9 was set, and say what you would set instead, with the daily cost of each option.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Three defects: the cut was a feeling rather than a comparison of two costs; nobody checked what a 0.9 meant before switching it on; and nothing sampled the auto-approvals for a month.

The costs set the line. A review costs $1 and a wrong approval $21, so a band should auto-approve only below $1 ÷ $21 = 4.8% wrong.

At 0.9, 150 of 200 auto-approve and 12 are ineligible: precision 92%, 8% wrong. A day is 12 × $21 = $252 of wrong approvals plus 50 reviews, $50: $302. Reviewing all 200 costs $200. The cut loses $102 a day against no automation at the step, about $3,060 over the unwatched month.

By band, 0.98 and up is 1 in 60 wrong, 1.7%, under the line; 0.90 to 0.98 is 11 in 90, 12%. A 0.98 cut auto-approves 60 a day with about 1 wrong, $21, and reviews 140, $140: $161 a day, precision 98.3%, recall 59 ÷ 168 = 35%.

Watching it is part of the price. At 1.7%, a window needs 10 ÷ 0.017 ≈ 590 reviewed approvals, a month of 60 a day read at 1 in 3: 20 reviews, $20 a day. That leaves $181 against $200, $19 a day saved, on one error in 60, which cannot tell 1.7% from 5%.

So set the cut at 0.98 with the 1-in-3 read, and let the read decide: more than about 28 errors in the first 590, 4.8%, switches auto-approval off. The 0.90 that suits $15 support replies in sampling review and confidence thresholds is wrong at $21.

Rubric

derives the break-even wrong rate, $1 ÷ $21 = 4.8%, and compares each band with it
prices the 0.9 cut per day, $252 of wrong approvals plus $50 of review, against $200 for reviewing everything
computes precision at the cut from the table: 92% at 0.9, 98.3% at 0.98
names the missing sample as a defect, with what it cost over the month
counts the cost of watching an auto-approved band before recommending one
states the evidence that would reverse the decision

Where answers go wrong

"Raise it to 0.95." Loses because it is another number chosen by feel: the table has no 0.95 band, and nothing about 0.95 refers to the $21 or the $1.

"The model is badly calibrated; fix the scores." Loses because calibration is not the defect: a perfectly calibrated 0.9 band is still wrong one time in ten, twice the 4.8% this step can afford. The cut has to come from the two costs.

"Keep 0.9 and sample the approvals from now on." Loses because a sample measures the 8% and stops none of it: the step still loses $102 a day, now with a chart of the loss.