The operating model and the catalogue · Owning a virtual employee

The review that found nothing

Find why six weekly reviews of the lead handler logged no change while referral replies went to the wrong person, and say what the review should have held.

critiqueeasyExercise · 4 min · 73 of 75

On this page

Six weekly reviews in a row logged "no change", and each was accurate about what it looked at. The leads that went wrong were in none of it. The lesson this practises is the weekly review.

The situation

The agency's lead handler answers about 400 leads a month, roughly 92 a week, from three sources: the website form (about 70%), the general inbox (15%) and the partner-referral form (15%). It looks each lead up in the CRM by email and drafts a reply. It is in its first weeks, on the launch setting: an account manager is meant to read 1 sent reply in 5, and the owner runs a 20-minute review every Thursday. Before launch about 9 leads a week matched no CRM contact. The first six review logs each read:

Sample: 0 wrong of 18, the first 18 replies sent on Monday. Escalations: 5, fine. Spend: $0.50 of tokens, under the $25 budget. Eval: not run, nothing changed. No change.

In week 7 the referral partner forwarded a reply addressed to her, about her own account. Since week 1 her form has put her own email in the email field and the client's in a new referred_email field, so every referral lead matched the partner's CRM record. Assume one wrongly addressed referral in ten is lost, a $1,500 first project.

The question

Find the defects in how these six reviews were run, and say what the week-2 review should have caught and changed.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Four defects, one per input, none needing more than 20 minutes.

The sample could not see the problem. The first 18 Monday replies are mostly website-form leads; a random 18 would hold about 3 referral replies a week, every one of them wrong. Draw the 18 at random across the week, at least 3 from each source. Even a sound sample reads 0 of 18 as "under about 3 ÷ 18 = 17%", not as zero.

The escalations were read as a count with no baseline. They ran at 5 from week 1 against the 9 expected. A shortfall is a finding: about 4 referral leads a week that would have matched nobody now matched somebody. Read it by source against the expected rate, and ask where the missing four went.

Spend was tokens only, which could never move on a wrong match.

"Eval: not run" is a defect, since the set should run weekly on a schedule, but it would not have caught this either: frozen cases never see a field the form started sending. That is a canary's job, one test referral through the real form every hour, checked for the right recipient.

In week 2, three wrong referral replies in a sound sample, or four missing escalations, would have shown it. The change goes to the connector, matching on referred_email, with referral replies held for a person until it ships. Not the prompt: the model wrote a sensible reply to the record it was handed.

Caught in week 2, about 14 referral replies a week × 5 weeks = 70 fewer reach the wrong person: 7 lost referrals, $10,500.

Rubric

shows the Monday sample held almost no referral replies, and replaces it with a random draw across the week with every source represented
reads 0 wrong of 18 as an upper bound of about 17%, not as no errors
treats 5 escalations a week against the expected 9 as a finding against the baseline, and asks where the leads went
says the scheduled eval would not have caught a field the form started sending, and names a canary or a per-source check that would
sends the fix to the connector, with referral replies held, not to the prompt or a gate
prices the delay: about 70 wrong replies and $10,500 at one lost referral in ten

Where answers go wrong

"The review was too short; read every reply." Reading all 92 sent replies a week at 3 minutes is 4.6 hours, about $600 a month, fourteen times the review's $43, and reading after sending stops nothing. A random 18 drawn by source would have caught the same fault in week 2. The defect was where the review looked, not how long it took.

"Run the eval set every week and this is caught." The frozen cases carry the form's old field layout, so the eval measures the model on inputs it already knows and never sees the new field. Running it is right; expecting it to catch an upstream change is not.

"Add an instruction telling the model to check the recipient." The lookup handed the model the partner's record, and nothing in a prompt can know that record is the wrong one. The input was the defect, so the fix belongs where the input is mapped.