The PM interview · The four questions

Execution, AI version

Segment a six-point pass-rate fall after a model update by ticket type, tenant and prompt version, and decide at which cut you stop and roll back.

casehardExercise · 3 min · 57 of 59

On this page

A model update lands on Wednesday and Monday's eval run is six points down. The interviewer hands over the run's breakdown and wants a decision today, which means deciding where the diagnosis stops; the execution lesson has the tree.

The situation

The drafting feature follows the vendor's "latest" model alias, and the vendor shipped an update on Wednesday. Monday's weekly run on the 200-case golden set scored 172 of 200, 86%, against 184, 92%, last week. The golden set and the help-centre index did not change this week. One other change did: on Thursday the largest tenant, 1,200 agents, moved to prompt v13, a tone change; every other tenant is still on v12. The run, cut three ways:

                               last week   this week
short replies (140 cases)         134         132
policy (40 cases)                  36          28
escalation (20 cases)              14          12
largest tenant (50 cases)          46          43    v13 this week
all other tenants (150 cases)     138         129    v12 both weeks
new fact-class failures             —           4

The done line is 92%, and the fact class blocks at any rate above 0.5%. The previous model version stays available for 30 days. Edit-before-send across 200,000 drafts a day has risen from 14% to 19% since Wednesday. Assume an edited draft costs an agent 20 seconds, at an assumed $30 an hour.

The question

Decide what you do today, and say at which cut in the diagnosis you stop and roll back.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Real: 12 cases net, 6 points, outside the ±3.8-point margin at 200 cases. The set and the index did not change, so this is neither new cases nor stale retrieval. Two changes remain: the model, for everyone, and v13, for one tenant.

Prompt version and tenant are one cut here, since v13 is one tenant. Its 50 cases fell 46 to 43, 6 points; the v12 cases fell 138 to 129, also 6 points. The same fall on both prompts clears the prompt; the model is the only change common to both. Rolling back v13 would fix nothing.

Ticket type: policy fell 36 to 28, 8 of the 12 cases; short replies and escalations lost 2 each. The update reads policy worse.

Stop here. Four new fact-class failures are 2% of the set on their own, against a 0.5% blocking rate, and no further cut — which policy, which tenant — changes the action: the last build that passed is last week's model. Roll back today, pin the version rather than follow the alias, re-run to confirm 184 before calling it fixed, and send research the 8 policy flips with their tickets.

Price the delay: 5 points of extra edits × 200,000 drafts is 10,000 a day; at 20 seconds each, about 56 agent-hours, roughly $1,700 a day, before any wrong fact an agent missed. The rollback costs only the update's gains, which nobody has measured.

What would change it: if the old model also re-runs near 172, something outside the list changed — the eval harness, a dependency — and the diagnosis returns to the first level.

Rubric

Checks the 6-point fall against the margin and rules out the set and the index before cutting
Compares the fall on v13 and v12 cases and concludes the prompt is not the cause
Locates the fall in policy tickets by ticket type
Stops at the blocking class and rolls back before the diagnosis is complete, saying why no further cut changes the action
Pins the model version and confirms the rollback with a re-run
Prices a day of delay in agent time

Where answers go wrong

"Roll v13 back for the largest tenant; it is the change we made." The v12 cases fell by the same 6 points, so the nearest change on your side is not the cause, and the model stays broken for every tenant.

"Find the root cause before touching production." Each day costs about 56 agent-hours and the fact class already blocks; the root cause is research's job on last week's model, not a reason to stay on this one.

"86% is within noise for a 200-case set." The margin is ±3.8 points, so 6 is outside it, and the 4 new factual failures block at any headline.