The measurement plan
Give experiments, the mix model and attribution one job each, order a year of tests by money at risk, and lay out a 12-month calendar for a $2M budget.
On this page
No single instrument measures marketing. Attribution is fast and biased, experiments are causal and slow, and a mix model sees every channel at once but with intervals too wide to settle a single channel's fate. A measurement plan gives each one the question it can answer, uses the experiments to correct the other two, and puts the tests on a calendar before the budget is spent rather than after.
Three instruments, three jobs
| Instrument | The decision it owns | How often | What it cannot do |
|---|---|---|---|
| Experiments: holdouts and geo tests | keep, grow or cut one channel | a few a year | cover every channel at once |
| Mix model | split next quarter's budget across channels | quarterly | judge one channel alone with confidence |
| Attribution | bids, creative, pacing, broken tracking | daily | measure cause |
The pairing follows from what each instrument is. Attribution divides credit, so it can compare two creatives inside a channel but not two channels. An experiment measures cause, but one channel at a time, over weeks, at a cost. A mix model estimates every channel's return curve from history, which is what an allocation needs, but its intervals are wide wherever the history was dull.
How experiments correct the other two
An experiment's result does two jobs after it is read. It becomes an incrementality factor for attribution: incremental orders divided by credited orders, over the same weeks. And it becomes a calibration point for the mix model: both open-source references document feeding experiment results into the fit, which narrows the interval of the channel tested.
Suppose the year's tests return these, against each channel's credited orders:
| Channel | Monthly spend | Credited orders | Credited cost | Factor | Incremental orders | Incremental cost |
|---|---|---|---|---|---|---|
| Paid social | $58,000 | 600 | $97 | 0.70 | 420 | $138 |
| Retargeting | $25,000 | 1,000 | $25 | 0.15 | 150 | $167 |
On credited orders, retargeting costs a quarter of what social does, and the next $10,000 would go to it. On incremental orders the ranking flips: social is cheaper per caused order. From then on, the weekly dashboard multiplies each channel's credited orders by its factor, so the daily numbers stay fast and stop being misleading. A factor goes stale when the channel changes, so retest when its spend moves by more than half, its targeting changes, or a year passes.
Ordering the tests: money at risk
A year has room for a handful of experiments, so they are ordered by how much money a wrong belief would cost: the channel's spend times the share of its credited orders you doubt. The doubt is an assumption, stated so it can be argued with: high for brand search, given eBay's result; high for retargeting, whose audience had already visited; high for a podcast, which produces no clicks to attribute at all.
Worked for a $2,000,000 budget, with each doubt assumed:
channel spend doubt money at risk
paid social $700,000 40% $280,000
podcast and audio $400,000 60% $240,000
retargeting $300,000 60% $180,000
brand search $200,000 80% $160,000
non-brand search $400,000 30% $120,000
Social and the podcast go first. One exception jumps the queue: a brand-search pause in a fifth of regions costs nothing to run, because the paused regions stop spending, so it starts in month 1. Non-brand search, the smallest risk, is left to the mix model this year and tested next year.
The calendar
The mix model is built in months 1 to 4 on the 30 months of weekly history the company has; its first output, with wide intervals, informs the month-6 re-split that sets the third quarter's budget, and the month-9 refresh carries all four test results as calibration. Each test has its size, its length and its decision rule written down before it starts, as the experiments lesson requires.
What the measurement costs, with each holdout's withheld exposure valued at the media price:
social holdout 20% x $700,000 x 8/52 weeks ≈ $21,500
podcast geo test media not bought ≈ $0 in cash
retargeting holdout 10% x $300,000 x 8/52 ≈ $4,600
brand-search pause media not bought ≈ $0 in cash
mix model 300 hours x $100 = $30,000
total ≈ $56,000, under 3% of the budget
The geo tests buy no media in their dark regions. What they give up is the margin on the sales those regions forgo, which the saved media covers exactly if the channel is at break-even.
Under 3% of the budget buys a defensible answer to whether the other 97% works. A company spending $2,000,000 without that answer is not saving the $56,000; it is paying it many times over in the channels it cannot rank.
The same structure scales down. At $200,000 a year there is room for one test, so it goes on the largest channel, and attribution scaled by that one factor runs everything else. The ai-pm course reaches the same conclusion for product work in when an A/B test is the wrong instrument: match the instrument to the question, and say what it costs.
In practice
To the board, every marketing number arrives labelled by the instrument that produced it: "credited", "incremental, from the March holdout", or "modelled, with this interval". Three labelled numbers that disagree are more useful than one unlabelled number that is wrong.
To agencies and channel owners, the plan is a contract: their channel will be tested in a named month, by a stated method, against a decision rule they saw first. Most agree, and the ones who resist a test of their own channel have said something about what they expect it to show.
The mistake that costs money is letting one instrument answer another's question: attribution splitting the budget, the mix model condemning a single channel, an experiment from two years ago still scaling this year's dashboard.
Check yourself
1. Budget allows two experiments this year, plus the free brand-search pause. Using the money-at-risk table, which two do you run, and what would make you swap one?
Social ($280,000 at risk) and the podcast ($240,000). Swap the podcast for retargeting if the podcast's doubt falls, for instance if a promo-code count shows most podcast buyers arrive with no other touch, or if retargeting's spend grows past about $400,000, where its money at risk overtakes the podcast's.
2. Social is credited with 600 orders a month at $58,000 and its holdout factor is 0.7; retargeting is credited with 1,000 at $25,000 with a factor of 0.15. Where does the next $10,000 go?
Social: $58,000 ÷ 420 ≈ $138 per incremental order against $25,000 ÷ 150 ≈ $167 for retargeting. Both are averages, so check the mix model's return curve for social before moving more than about a fifth of its budget in one quarter.
3. The mix model says the podcast returns 0.8 orders per $1,000, with an interval of −0.5 to 2.1. The geo test says 1.9, from 1.2 to 2.6. Which number do you plan with?
The geo test's: it is causal and its interval is about half as wide. The model's interval was wide because podcast spend barely varied in the history. Feed the test into the model as a calibration point, plan next quarter on 1.9, and retest when podcast spend changes by more than half.