Discovery · Interviewing and prototypes

The interview that proved nothing

Find the three leading questions in a transcript that manufactured a validated snapshot, and rewrite them as story questions that produce evidence.

critiqueeasyExercise · 3 min · 10 of 59

On this page

A 25-minute call with a support lead ended with three claims marked validated. The transcript supports none of them, and the exercise is to find the three questions that did the damage and rewrite them.

The situation

A PM is deciding whether the drafting branch belongs on the tree and books 25 minutes with the support lead at a 40-agent tenant. The notes record six questions and their answers:

"We're building an AI assistant that drafts replies — would your agents use something like that?" — "Sure, probably."
"How much time do you think it would save per ticket, maybe a minute?" — "A minute sounds right."
"Do your agents find the macros hard to use?" — "Some do."
"Tell me about yesterday's queue at 4 pm." — "Around 90 tickets, mostly password resets and refunds. We carried maybe 40 overnight."
"Wouldn't they prefer a draft to picking from 200 macros?" — "I think so."
"If we shipped this next quarter, would you be a reference customer?" — "Happy to."

The PM's snapshot reads: "Validated: agents want AI drafts; saves about 1 minute a ticket; agents prefer drafts to the macros. Reference customer agreed." The tree now carries the branch with the lead's tenant counted as evidence for it.

The question

Which three questions produced the snapshot's three claims, and how would you rewrite each so that the same 25 minutes returned evidence instead?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Questions 1, 2 and 5 each manufactured one claim. Question 1 names the product and asks for a prediction, so "sure, probably" became "agents want AI drafts". Question 2 puts the number in the question, so "a minute sounds right" became "saves about 1 minute". Question 5 has its answer built in, so "I think so" became a preference for drafts over macros nobody tested.

The rewrites, as continuous interviewing sets the rules: past tense, one instance, no product in the sentence, then silence.

Question 1 becomes "Tell me about the last ticket you answered yourself — what did you do between opening it and sending?" It finds the steps of a reply, and the macro search inside it, without a product in the room.

Question 2 becomes "How long did that one take, from opening to send? And the one before it?" Two remembered instances give a range from real events; "maybe a minute" gave the PM's own guess back.

Question 5 becomes "Walk me through how you picked the macro on that refund ticket — what did you type, and how many did you look at?" It finds out whether the macro is the pain or the habit, which is the difference between a drafting tool and a search box.

Question 4 stays: it is the only story question and the only answer with numbers in it, 90 at 4 pm and 40 carried. Question 6 goes; a reference offer is politeness. The snapshot is rewritten to what was heard: 90 tickets at 4 pm, about 40 carried overnight, password resets and refunds dominate, reply time unknown, macro pain unknown, nothing validated. The branch keeps its place on the tree, with this tenant counted for the overnight pain only.

Rubric

Identifies questions 1, 2 and 5 as the leading ones and ties each to the specific claim it produced in the snapshot.
Rewrites each as a past-tense, single-instance story question with no product named in it.
Keeps question 4 and says why it is the only one that returned evidence.
Rewrites the snapshot so it claims nothing the transcript does not support, and says what is still unknown.
Leaves the tree honest: the tenant counts for the overnight pain, not for the drafting branch.

Where answers go wrong

"Make question 2 quantitative: ask them to rate the time saved from 1 to 10." A number on a hypothetical is still a hypothetical; the lead has never seen a draft, so the scale measures politeness with more precision.
"The interview was fine — the lead agreed to be a reference, which is the strongest signal there is." A reference offer costs the lead nothing today. Nothing about the drafts was timed or counted, and the one answer with numbers in it came from the one question the critique did not notice.
"Rewrite all six." Question 4 is right, and question 3, though leading, produced no claim in the snapshot. A critique that cannot rank the defects will not get another 25 minutes with the lead, and the reviewer scores the ranking, not the volume.