Evals with the customer's definition of good · The eval set

Precision or recall for the fraud queue

Decide whether a fraud flag feeding an investigations unit that opens 18 files a day is tuned for precision or recall, and set the operating point.

casemediumExercise · 4 min · 42 of 64

On this page

A fraud flag is the first output in this deployment whose errors land on two different groups: investigators who open the wrong files, and honest claimants whose payment waits. Which of the two the threshold protects is a decision, and the eval has to be built to inform it.

The situation

Phase 2 of the insurer's deployment, scoped after the triage pilot renewed; the pilot's scoping doc had ruled fraud scoring out. The sponsor wants a flag that refers suspicious claims to the special investigations unit (SIU). The floor handles 1,200 claims a working day, and the SIU estimates from claims later proved fraudulent that about 1% are fraud: 12 a day. Its three investigators open 6 referrals each a day, 18 in all. Today the floor routes about 5% of claims, 60 a day, to fraud review; supervisors pass 18 of them to the SIU by judgement, and 1 in 3 of those is confirmed, so 6 frauds a day are caught. A referred claim's payment is held until the SIU clears it, about 10 working days, and held honest claims are the complaints team's largest category. The team built a fraud eval from SIU outcomes, 100 confirmed frauds and 300 claims cleared on investigation, reweighted to the live 1%. It gives three operating points: refer 18 a day at 50% precision, 30 a day at 35%, or 45 a day at 24%.

The question

Decide whether the fraud flag is tuned for precision or for recall, choose its operating point, and say what you tell the head of the SIU.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Precision, at the SIU's capacity, with recall reported beside it. The queue is bound by 18 investigations a day; a referral beyond the eighteenth is not investigated, it is an honest claimant's payment held for nothing.

Work the points. At 18 a day and 50% precision the SIU confirms 9 a day: recall 9 ÷ 12 = 75%, up from today's 50%, and held honest claims fall from 12 a day to 9, with no new staff. At 30 a day and 35%, the flag would catch 10.5 if all 30 were opened; with 18 opened the other 12 wait, and 19.5 honest claims a day are held. At 45 and 24%, 10.8 frauds and 34 honest claims held. Past capacity, recall is bought with complaints.

To the head of the SIU: "At your 18 a day, the flag should take confirmations from 6 to about 9. Going to 30 needs two more investigators, buys about 1.5 frauds a day, and holds about 10 more honest claims. Whether 1.5 frauds a day pays for two investigators is your number." That is also what would reverse the decision: if it does, the operating point moves to 30.

Two caveats go on the page, as building the eval set would insist. The precision figures needed the reweighting to 1%, and hold only as far as it does: the 300 cleared claims were suspicious enough to investigate, not a sample of the 1,188 honest claims a day, so check them on a week of live flags. And recall counts only fraud the SIU has found, so frauds nobody referred are missing from it, and the true recall is lower. The 200-case triage set cannot help: at 1% it holds two frauds.

Rubric

names the SIU's capacity of 18 a day as the constraint that decides between precision and recall
works each operating point into frauds caught and honest claims held per day, with the arithmetic
compares the chosen point with today's 6 caught and 12 held
puts the investigators-for-recall trade to the SIU head in his units, and leaves the value judgement with him
notes that precision measured on an eval enriched with fraud must be reweighted to the live rate
says why the 200-case triage set cannot measure a 1% label

Where answers go wrong

"Recall, because a missed fraud costs real money and a false referral only costs an investigator's time." Loses because a referral beyond the 18 the SIU can open costs no investigator time at all: it costs an honest claimant ten working days of held payment, and the extra recall is never realised.

"Refer 45 a day and let the investigators prioritise." Loses because the queue now does the threshold's job by hand, and 24% precision becomes 34 held honest claims a day, nearly three times today's 12, landing in the complaints team's largest category.

"Use the triage eval set; it already has the customer's labels." Loses because 1% of 200 is two frauds, so any precision or recall on it is a coin toss, and its labels are escalation decisions, not SIU outcomes.