The PM interview · The four questions

The behavioural answer that hid the number

Find the number missing from a STAR answer about summaries that invented order numbers, and rewrite its result line so an interviewer can score it.

critiqueeasyExercise · 3 min · 59 of 59

On this page

A candidate's STAR answer reads well and scores poorly, and the reason is one line. The behavioural lesson says where each number goes; this answer left the most important one out.

The situation

The prompt was "tell me about a time you shipped something the model got wrong". The candidate's answer, as given:

I owned ticket summaries for a support tool used by 8,000 agents, about 160,000 summaries a day. A month after launch, agents began reporting that some summaries quoted order numbers that were not in the ticket. I pulled 300 summaries from one week and found 21 with an invented order number. I worked with the engineers to make the model copy order numbers only from the ticket's order field, added the 21 as golden-set cases, and made "invented identifier" a class that blocks any release. As a result, the summaries became much more accurate, agents trusted them more, and other teams adopted the blocking class for their own features.

The interviewer's first follow-up is the one every result line like this one gets.

The question

Find the number the result line is missing, and rewrite the line so it can be scored.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

The before is there: 21 of 300 is 7%. The after is missing, and "much more accurate" is where the interviewer interrupts: "from 7% to what, on how many?" Without the after, the story proves the candidate found a problem, not that they fixed it.

The result also hides the scale the answer itself gave. 7% of 160,000 summaries a day is about 11,000 a day carrying an invented order number. That is the figure that makes the story matter, and it belongs in the result as the before.

"Agents trusted them more" is a feeling. The behaviour that would show it is a count: the share of tickets where agents open the summary, or the reports filed per week. If the candidate never measured it, the clause goes.

The rewrite, assuming the candidate re-ran the same check and these are their real figures:

On a fresh sample of 300 the next week, 1 summary had an invented order number: 0.3% against 7%, or about 500 a day instead of 11,000. The blocking class has since stopped two prompt changes before release.

The after is outside the noise: on 300 summaries the margin around 7% is about ±3 points, and 7% to 0.3% is well past it.

If the candidate did not re-run the check, the honest line says so and names the measure: "I did not re-sample, which I would now; agent reports stopped within a week." An invented 0.3% survives the first follow-up and fails "how did you sample it?"

Rubric

Names the missing after, on the same measure as the 7% before
Converts the 7% to a daily count at 160,000 summaries a day
Rewrites the result with before, after, sample size and unit
Replaces "trusted them more" with a behaviour count, or removes it
Says a number the candidate does not have must not be invented, and gives the honest alternative line
Checks the after against the sample's margin

Where answers go wrong

"Add a percentage: accuracy improved by 95%." A relative change with no base and no sample; the interviewer asks "95% of what?" and the missing number is still missing.

"The action is too long; cut it." The action is the strongest part — a fix, cases added, a blocking class. The defect is in the last line.

"Lead with the result instead." Order is not the defect; a result line with no number scores nothing wherever it sits.