What AI changed: content, search, research · Research

Critique this synthetic-research readout

Find the defects in a synthetic panel's market size and price, decide which findings may inform an add-on launch, and rule on two sales hires.

critiquehardExercise · 3 min · 44 of 60

On this page

A synthetic panel has produced a market size, a price and a hiring case in two days. Most of the readout is usable; the parts the co-founder wants to act on are not.

The situation

The clinic scheduler from module 1 has 400 paying clinics at $99 a month, in a segment of 40,000 small clinics. It is considering a no-show deposit add-on at $49 a month, $588 a year. A research vendor delivered a synthetic readout for $1,500: 1,000 language-model personas of clinic owners, built from public demographic data, answered a 20-question survey. Findings: 62% interested; 24% "would definitely buy" at $49; median willingness to pay $58; three objections, ranked; "deposit" preferred to "card hold" three to one. The last slide: 40,000 × 24% × $588 ≈ $5.6M a year, priced at $59. The co-founder wants two sales reps on that number, at $150,000 a year loaded each and, by the course's constant, 30 closed deals a rep a year. Rerunning with one question reworded gave 31% "definitely". The vendor has never compared its panels with human respondents in this category. A human panel of 300 clinic owners would cost about $9,000 and three weeks.

The question

Find the defects in the readout, decide which of its findings may inform the add-on decision, and rule on the two reps.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The $5.6M comes off the slide. It sizes a market from synthetic stated intent, the one use the evidence does not support. Argyle et al. matched vote shares using backstories taken from real respondents, on one of the most written-about opinions there is, and still missed a topline by six points. These personas come from public data, and the category is new. Brand, Israeli and Ngwe found fine-tuning on human results helped with new features in a known category, not with new categories. The vendor's own rerun moved 24% to 31% on one reworded question, Bisbee et al.'s wording sensitivity: 7 points is 2,800 clinics, about $1.6M of "market" from wording. The $58 and the $59 fail for the same reason.

The reps fail even if the number were right. Two reps close 60 deals a year, 60 × $588 ≈ $35,000 of revenue a year against $300,000 of cost. A $588 add-on is a self-serve sale to existing clinics.

What may inform the decision: the objections, as an interview guide; "deposit" against "card hold", as wording to pre-test; and the questionnaire, now debugged.

Instead, offer the add-on in-product to the 400 current clinics at $49 for a month and count attaches. If 10% attach, that is 40 × $588 ≈ $23,500 a year and evidence no persona can give. Spend the $9,000 on the human panel, price questions included, before anyone sizes the market.

Hire no reps on this readout. What would reopen the question: a human-measured attach rate, and a contract value large enough for a rep to pay for.

Rubric

removes the market size and the price, citing why synthetic stated intent cannot size or price a new category
turns the 24% to 31% rerun into a number of clinics or revenue
tests the rep hire on rep economics, independently of the readout
names the findings that may inform the decision and how each is used
proposes a behavioural or human test with a cost and a result that would change the decision
cites at least one study on synthetic respondents with its limit, rather than the vendor's own claims

Where answers go wrong

"Synthetic research is not real research; ignore the whole readout." It throws away the objections, the wording result and the debugged questionnaire, which are what a synthetic panel is good for, and offers nothing in their place.

"Halve the 24% for optimism: $2.8M still justifies two reps." A correction factor with no human calibration in this category is a guess applied to a guess, and the reps lose money at any market size, since 60 deals at $588 cannot cover $300,000.

"Rerun it with 10,000 personas to tighten the estimate." More personas shrink the sampling error and leave the bias where it is; the 7-point swing came from wording, not sample size, as the lesson on synthetic research sets out.