The PM interview · The four questions

Product sense, AI version

Decide whether the support tool should answer end customers directly, with the failure mode as a weekly count, the metric, and what the assistant will not do.

casemediumExercise · 3 min · 56 of 59

On this page

The interviewer turns the drafting feature around: the same model, but nobody reads the reply before the customer does. The round is scored on whether you notice that the 92% was measured with an agent in the loop; product sense works the shape of the answer.

The situation

"Our support tool drafts replies for agents today. Should we add an assistant that answers the end customer directly, in the tenant's help widget, before a ticket is opened?" What you know: the tool sees 2.5 million tickets a week across all tenants. On the 200-case golden set, 92% of drafts are acceptable, and half a point of the 8 that are not, one draft in 200, is wrong on a fact — a price, a policy, a date. Agents edit or discard most wrong drafts before sending. 70% of tickets are short: order status, delivery dates, password resets. A wrong reply that reaches a customer costs a 20-minute escalation, and tenants' contracts carry a clause on factual errors in customer-facing text. The widget today offers article search. Tenants pay $5 a seat a month for the drafting add-on, and a tenant that needs fewer agents eventually buys fewer seats. You have 40 minutes.

The question

Decide whether to build it, for which tickets, measured by which metric, and say what the assistant will not do.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Restate: should a model reply to customers with no agent between them, and for which tickets? The user is the tenant's customer; the buyer is the tenant's support lead.

The number: the 92% was measured on drafts an agent reviews. Remove the agent and the half-point factual error rate goes straight out. On the short 70% alone, 1,750,000 tickets a week × 0.5% is about 8,750 factually wrong answers a week to customers, each one inside the contract clause. As asked — answer anything — no.

Options. Reject the general assistant on that count. Reject no assistant, because part of the short slice is a lookup, not a composition: order status and delivery date sit in the order record, which the model can quote rather than write. Choose a narrow assistant for those two intents, launched to 5% of one tenant's widget traffic.

It will not answer refund, billing or policy questions; will not promise or take an action such as a refund, a cancellation or an address change; will not answer without showing the record it read. Every answer offers a person and carries the transcript into the ticket.

Metric: resolutions without a human touch, counted only when the customer opens no ticket within 7 days. Not deflection, which rises when the assistant stops customers reaching a person. Guardrails: a new golden set of at least 600 cases for the two intents, stratified by tenant, with factual errors under 0.5%, and complaints per thousand sessions.

Pricing: success means fewer seats, so propose a per-resolution price before the beta widens.

What would change it: factual errors above 0.5% end the beta; a 7-day no-reopen rate at or above the agents' 0.90 earns a third intent.

Rubric

Notices the 92% was measured with an agent reviewing, and recomputes the error without one as a weekly count of wrong customer-facing answers
Narrows to intents whose answer is a lookup from a record, rejecting both the general assistant and no assistant with a reason each
States at least three things the assistant will not do, including taking actions and answering policy questions
Names an outcome metric with a 7-day reopen check, and rejects deflection as a metric that rises when the product gets worse
Names the thresholds that end or widen the beta
Notices that the per-seat price works against the assistant's own success

Where answers go wrong

"Yes — customers want instant answers, and the model already scores 92%." The 92% includes an agent catching the wrong drafts; take the agent out and the catch goes with it, and the answer never counts the roughly 8,750 factually wrong answers a week.

"Measure deflection: the share of widget sessions that never become tickets." A customer who gives up is deflected, so the metric rewards an assistant that is hard to get past.

"No — a model should never talk to customers." A refusal with no number, which misses the slice where the answer is a record lookup and a new golden set can measure the risk before anyone is exposed to it.