Synthetic research and its validity limits
Judge a synthetic-research readout: use it to sharpen what you ask real buyers, and keep it out of market sizing, pricing and willingness to pay.
On this page
A synthetic panel asks a language model to answer a survey as hundreds of described people, "a clinic owner in Ohio with two practitioners and a paper diary", and returns a readout in an hour. It looks like research, with percentages, segments and quotes. Whether it is depends on the use: the evidence supports deciding what to ask real people with it, and does not support sizing a market, setting a price or estimating what anyone will pay.
What a synthetic panel is
Each persona is a short description in a prompt; the model answers the survey in that voice, and the answers are tabulated as if each persona were a respondent. Assume a vendor charges $1,500 for 1,000 personas and two days. A human panel of 300 clinic owners at about $30 a completed survey is $9,000 and three weeks.
The answer is a draw from the text the model learned, conditioned on the description: what people described like that have tended to write. That is close to the truth for opinions that are heavily written about and stable, and far from it for a decision nobody has written about, such as paying $49 a month for a product that does not exist. Answers also vary from run to run and with small changes of wording, for the reasons in sampling and determinism.
The evidence for
The result most often cited is Out of One, Many: Using Language Models to Simulate Human Samples (opens in a new tab) (Argyle et al., Political Analysis, 2023). The authors gave a language model the demographic and attitude backstories of real respondents to the American National Election Studies of 2012, 2016 and 2020, and asked how each would vote. The simulated two-party vote shares came close: 39.1% against 40.4% in 2012, 43.2% against 47.7% in 2016, 47.2% against 41.2% in 2020, with correlations between simulated and reported votes of 0.90 to 0.94. They called the property "algorithmic fidelity".
Read its limits before borrowing its conclusion.
- The backstories came from a real survey. The model was handed each respondent's characteristics and asked for one more answer; the survey, not the model, produced the population.
- The question was among the most written-about there is. US presidential vote choice by party, ideology and church attendance fills decades of text. A clinic owner's willingness to pay for a deposit feature does not.
- Even so, one topline missed by 6 points, in 2020, and the fit was weakest for pure independents, the least predictable voters, who in a market are the undecided middle a marketer most needs to read.
- The authors say the study is restricted to one domain, and call for work to measure the extent and limits of fidelity in others.
The evidence against, and where it is thin
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models (opens in a new tab) (Bisbee, Clinton, Dorff, Kenkel and Larson, Political Analysis, 2024) asked a chat model, in persona, for feelings toward 11 social and political groups. The averages tracked the human survey's closely. Everything underneath was worse: the synthetic answers varied less than people's, regression coefficients often differed significantly from those in the human data, small wording changes shifted the distribution, and the same prompt gave significantly different results three months apart.
The closest study to marketing is a working paper, not yet peer-reviewed: Using LLMs for Market Research (opens in a new tab) (Brand, Israeli and Ngwe). Simulated willingness to pay replicated findings for existing product features, and after fine-tuning on earlier human survey results it predicted willingness to pay for new features in the same category. Fine-tuning did not carry over to unrelated categories, and segment differences may need separate models. What made it work was human data from the same category, exactly what a new product lacks.
Where the evidence is thin: we found no peer-reviewed study showing synthetic respondents sizing a market or pricing a new category correctly. The accuracy claims we found for marketing use come mostly from vendors of synthetic panels, usually measured on questions whose answers were already published.
Why sizing and pricing are the line
Worked, with the clinic scheduler from targeting: the beachhead arithmetic: 40,000 small clinics, $99 a month. A synthetic readout says 18% of clinic owners would definitely switch, and the slide turns that into 40,000 × 18% × $1,188 = $8.6M of annual revenue. Three things ride inside that figure.
Instrument error. A 6-point miss, the size Argyle et al. saw on the best-documented question there is, makes 18% anything from 12% to 24%: $5.7M to $11.4M. Each point is 400 clinics and about $475,000 a year.
False precision. The readout's margin of error, ±2.4 points from √(0.18 × 0.82 ÷ 1,000) × 1.96, describes the sampling of 1,000 personas, not the error of the instrument. Ten thousand personas would shrink it and leave the bias where it was, and compressed variance makes segment differences look surer than they are.
Stated intent. Not everyone who tells a human survey "definitely" goes on to buy. A synthetic panel inherits that overstatement, adds its own error, and in a new category has no human data to correct either.
Against that, the beachhead arithmetic, from an assumed win rate the first sales will test: 40,000 × 2% × $1,200 = $960,000. A market size feeds hiring, fundraising and targets, so an error in it is expensive and found late, after the reps are hired; hence never, not "with care". Price and willingness to pay fall on the same side, and the one study that got close needed the category's own human results.
Where it helps
The same readout is useful when its output is a question for people, not an answer.
- Drafting questions. Run the questionnaire through 50 personas to find items that are ambiguous, double-barrelled or answered identically by everyone, and fix them before the $9,000 panel.
- Generating hypotheses and objections. A list of reasons a clinic owner might refuse, taken into interviews; continuous interviewing is where they get tested.
- Pre-testing wording. Which of ten headlines confuses, before a live test spends traffic on it.
- A concept shortlist. Which 3 of 12 concepts go to the human panel. A wrong cut is cheap, because people see the survivors.
If synthetic answers are to stand in for a recurring question, validate first: run synthetic and human panels side by side on questions whose answers were unknown at the time, agree a tolerance in advance, such as 3 points on each topline and the same segment ranking, and repeat every wave, because results drift. The human panel is the evaluation set, as in evals as the unit test of LLM work, and the permission covers only questions inside it.
In practice
To a vendor of synthetic panels: "Show us your results beside a human panel's, in our category, on questions whose answers were not known when you ran them." Published answers may have been in the training text.
To the board or a co-founder: a synthetic readout appears as "what we will ask customers" or "which concepts we will test", never as a share, a size or a price; those come from the beachhead arithmetic, a pricing test or a human panel.
The mistake that costs money is putting a synthetic "would buy" share on a market-sizing slide. At 40,000 clinics each point is about $475,000 a year, a 6-point instrument error is $2.9M either way, and the hiring plan built on it is paid for in salaries before anyone measures the market.
Check yourself
1. You have 12 product concepts and room for 3 in next month's 20 customer interviews. A synthetic panel would rank them for $300 in a day. Use it?
Yes. It informs a cheap, reversible choice whose errors the interviews correct. Ask it for each concept's objections too, and put one concept it ranked low into the interviews as a check.
2. A synthetic readout gives a median willingness to pay of $140 for the scheduler, and the pricing lead wants to move from $99 to $129. Decide.
Not on this. The evidence supports willingness to pay only after calibration on human answers from the same category, and none was done. Price is cheap to test with real buyers: offer $129 to a random half of new sign-ups for a month and compare paid conversion and revenue per visitor. The readout may suggest price points; the test decides.
3. A vendor shows its synthetic panel matched your last human panel within 3 points on 20 questions. Do you drop the $9,000 annual human panel?
No. Those answers existed before the synthetic run and may have shaped it, and Bisbee et al. found results moving with wording and over three months. Keep the human panel as the calibration set and for every new question; use synthetic runs to pre-test the questionnaire, which may shorten the human survey.