Human in the loop · Gates and review

Gate placement for a refund flow

Decide which steps of an eight-step membership cancellation stop for a person, on what condition and clock, and what the review costs a day.

designmediumExercise · 3 min · 29 of 75

On this page

An online retailer wants membership cancellations automated end to end, and the step that most needs a person is not the one that moves money. Where the gate goes decides both the daily review bill and the worst thing one misread request can do.

The situation

The online retailer this module uses in place of the agency sells a paid membership at $120 a year, and cancellations arrive at 40 a day. The proposed flow:

Read the request and classify it (model).
Look up the membership and payment history (code, read only).
Compute the refund: unused full months × $10, capped at $120 (code).
Turn off auto-renewal (code; turned back on in one click).
Refund the computed amount to the original card.
Downgrade benefits at the end of the paid period (code; reversible).
If the customer asks, delete their stored addresses and saved card (cannot be restored).
Send the confirmation email.

Assume the model misjudges eligibility on 0.5% of refunds, which average $60. One customer in ten asks for deletion, 4 a day, and the model misreads 5% of those, taking "remove my card" for "delete everything", at $200 each in a lost customer and support time. Confirmation emails are wrong 2% of the time, at $5 each. A reviewer takes 2 minutes at a loaded $30 an hour.

The question

Place the gates: say which steps stop for a person and on what condition, who approves, on what clock, and what the review costs a day.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

One gate, at step 7, on every deletion. The money step does not get one.

The test from approval gates is error rate × cost if wrong against the $1 a 2-minute review costs, applied to steps that cannot be undone. Steps 1 to 4 and 6 are internal or undone in a click. Step 5 moves money, but code computes and caps the amount, so the model only judges eligibility: 0.5% × $60 = $0.30 on average, and even a full $120 refund is $0.60. Gating step 5 would spend $40 a day to prevent $12 a day. Sample it instead. Step 8 is irreversible but cheap, 2% × $5 = $0.10: sample. Step 7 cannot be undone and is misread often: 5% × $200 = $10 of expected loss per deletion, ten times the review. Gate every one.

The gate has four states: pending, approved, rejected, expired. The support lead approves, seeing the customer's exact words beside the action. Nobody is harmed by a deletion that waits a day, so the clock is one working day, and expiry goes to the owner, never to delete. Four a day at 2 minutes is 8 minutes and $4 a day, against $40 a day of expected loss removed.

Order matters. Steps 1 to 6 run at once; step 8 confirms the cancellation and refund and says the deletion is under review, and a second email confirms it after approval, so nobody is told data is gone while it is pending.

What would change it: an eligibility error above 0.83% ($1 ÷ $120) would put the largest refunds over the line.

Rubric

applies error rate × cost against the $1 review cost to each irreversible step, with the arithmetic shown
leaves the refund step ungated because code fixes and caps the amount, and samples it instead
gates every deletion, naming irreversibility and the 5% misread as the reason
specifies the gate: its states, the approver, a clock, and expiry that escalates rather than deletes
sizes the queue at 4 a day and 8 minutes, against the expected loss it removes
keeps the confirmation email from claiming a deletion that is still pending

Where answers go wrong

"Gate the refund, because money is the risk." Loses because code computes and caps the amount; a person checking every refund spends $40 a day to prevent $12 a day of loss, while the deletion, the one step nothing can undo, runs unreviewed.

"Gate every step for the first month, then relax." Loses because 40 cancellations × 8 steps is 320 approvals a day, almost 11 hours; reviewers learn to click approve, and the misread deletion arrives inside that habit.

"Gate the deletion, and approve it automatically if nobody answers in 24 hours." Loses because the timeout turns the gate into a delay: every unread deletion runs at hour 24, which is exactly the case the gate exists to stop.