Running it: evals, cost and security · Evals and monitoring

The dashboard with only error counts

Find why a green dashboard missed 24 mishandled partner leads, and choose three rate alarms with thresholds sized from the lead handler's own volume.

critiqueeasyExercise · 4 min · 60 of 75

On this page

The lead handler's dashboard was green for three weeks while one source's leads got the wrong reply. Nothing on it was broken; it measured the wrong things, the failure described in drift and silent failure.

The situation

The agency's lead handler takes 400 leads a month from three sources: the website form, the general inbox and a partner-referral form, which sends about 60 a month, 2 a day. For each it writes a CRM row and, within five minutes, sends a reply: leads in the under-$5k budget band get the self-serve package page, the rest a link to book a call. Normally about a quarter get the self-serve reply. It is the handler's second month, still on the launch review: an account manager reads one sent reply in five. The dashboard has four tiles: runs per day, errors per day, average run time, and spend this month, about $2.

For 21 days every tile looked normal, and every lead was done inside five minutes. On day 22 an account manager noticed that no partner referral had booked a call. On day 10 the partner form had replaced its budget options with new labels; the connector mapped any label it did not recognise to the default band, under $5k, so all 24 partner leads since then got the self-serve reply. Assume one call-worthy lead in five becomes a $1,500 project.

The question

Find what the dashboard cannot see, and say which three rate alarms you would add, with the threshold for each.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Every tile counts something that stayed normal while the output went wrong. Runs per day counts arrivals, not what happened to them. Errors count exceptions, and a default value raised none. Run time and spend measure effort. Nothing on the board describes the decision the automation makes, so a jump from 25% self-serve to 100% for one source was invisible. About 18 of the 24 should have been offered a call; at one in five, three or four projects, about $5,400 of pipeline, went to a self-serve page.

Three alarms, each per source, each to the account manager who owns the role:

Defaulted or empty share, per input field. The connector logs every fallback. Baseline under 5%; alarm above 20% of a source's last ten leads. The relabelled budget field defaulted every partner lead from day 10, so this fires on day 11, at the third.
Self-serve share, over a source's last 12 leads. At 2 partner leads a day that is six days, normally 3 self-serve. Alarm at 8 or more of 12, which chance alone produces under 0.3% of the time. It also catches a scoring change no relabelling caused.
Calls booked per source per fortnight. Partner referrals normally book about 8. Alarm below 2; a quiet fortnight rarely falls that far by chance, and this alarm watches the outcome, whatever broke upstream.

One smaller fix: make the connector refuse a budget label it does not know, so the lead goes to the person queue with a reason instead of taking the default.

Rubric

says why each of the four tiles stayed normal
states that nothing on the dashboard describes the automation's decision
adds a defaulted-or-empty alarm per input field and source, with a threshold
adds an outcome-share alarm per source, with a window sized from the volume
adds one alarm on a downstream result, such as calls booked
puts the 12 days into leads or pipeline lost

Where answers go wrong

"Alert when daily lead volume drops." It never dropped. Thirteen leads a day kept arriving and finishing on time; they got the wrong reply. A volume alarm watches the input, and this failure was in the decision.

"Alarm when the daily self-serve share moves by 5 points." At 13 leads a day the share's standard error is √(0.25 × 0.75 ÷ 13) ≈ 12 points, so a 5-point band fires most days and gets muted within a week. The window has to fit the volume, which is why the strong answer counts each source's last 12 leads.

"The account manager already reads one sent reply in five; read more." She read about five partner replies in those 12 days, and each was a correct self-serve reply for a lead whose row said under $5k. Reading outputs checks the reply against the row, never the row against what the partner actually sent. Even the gate the agency adds in month 3, reading every reply before it sends, would have approved all 24.