Human in the loop · Gates and review

The sampling rate for support replies

Estimate how many auto-sent support replies a reviewer reads each day, as 1 in N with minutes and dollars, and how review changes after a prompt change.

estimatehardExercise · 3 min · 30 of 75

On this page

Above the confidence cut, an online retailer's support replies go out with nobody reading them first. How many someone reads afterwards is a number with a cost on each side: reviewer hours on one, days of a broken prompt running unnoticed on the other.

The situation

The online retailer this module uses in place of the agency auto-sends 900 support replies a day, seven days a week: the 30% of tickets whose draft scores 0.90 or more. At that cut the labelled set puts 30 of the 900 wrong, about 3.3%. Assume a wrong reply costs $15, a reopened ticket with 10 minutes of a person and a $10 credit, and that reading a sent reply takes a reviewer 2 minutes at a loaded $30 an hour. Assume a review window needs about 10 errors at the normal rate before a doubled rate shows. The automation changes four times a month, through prompt edits, a model update or a renamed upstream field, and assume one change a month doubles the wrong rate. A weekly read finds a doubling about three days later than a daily read would.

The question

Set the standing sampling rate, as 1 in N with reviewer minutes a day, and say what review does on the day after a change.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Sample, do not gate: 30 ÷ 900 × $15 = $0.50 of expected loss a reply, under the $1 a review costs. The method is the one in sampling review and confidence thresholds.

The window needs 10 ÷ (30 ÷ 900) = 300 reviewed replies.

Daily window: 900 ÷ 300 = 3, so 1 in 3, 300 a day, 600 minutes, $300 a day.
Weekly window: 6,300 ÷ 300 = 21, rounded to 1 in 20 for margin: 45 a day, 90 minutes, $45 a day.

A doubled rate costs an extra 30 × $15 = $450 a day, so three days later is $1,350 an incident, one a month. The daily window costs $255 a day more, about $7,650 a month. Standing rate: 1 in 20, 45 replies and 90 reviewer minutes a day.

The incidents come from changes, so the review follows them. For the day after each of the four monthly changes, read at the daily rate: 1 in 3, 300 replies, $255 more than a standing day, about $1,020 a month. A doubling caused by a change is then found as a daily read finds it, three days sooner than the weekly read, so $1,020 of extra reading saves the $1,350 incident, and the month's review totals about 30 × $45 + $1,020 = $2,370.

Two things would reverse it. A lower wrong rate needs more reviews, not fewer: at 1%, the window needs 1,000, which is 1 in 6 weekly. And above $30 a wrong reply, 3.3% × cost passes $1, so the replies need a gate: the confidence cut moves up instead.

Rubric

tests gate against sample first: 30 of 900 × $15, $0.50, against the $1 review
converts the 10-error rule into a sample of about 300 per window
prices at least two windows in reviewer minutes and dollars
prices a doubling per day and per incident, and compares it with the extra review a shorter window costs
raises the rate after changes and costs it
names what would reverse the answer: a lower wrong rate, or a wrong reply costing more than $30

Where answers go wrong

"1 in 100, which is what everyone uses." Loses because 9 replies a day is 63 a week holding about 2 errors, so a doubling to 4 looks like an ordinary week and runs at $450 a day until a customer's complaint finds it.

"Read every reply until we trust it." Loses because 900 × 2 minutes is 30 hours, $900 a day, to stop $450 a day of errors; that is a gate, and on these costs a gate on these replies loses $450 a day.

"1 in 20, because errors are about 3% and that feels proportionate." Lands on the right number for the wrong reason: the same reasoning at 1% would shrink the sample when it has to grow to 1 in 6.