Measurement · Mix modelling

Read this MMM output

Decide whether to move $80,000 a month from social to search on a mix model's coefficient table, reading each interval against zero and break-even.

caseeasyExercise · 3 min · 35 of 60

On this page

A first mix model has arrived, and the CMO has already read it as a verdict. The table supports a narrower claim, and marketing mix modelling shows how to read one.

The situation

A direct-to-consumer brand's analyst delivers its first marketing mix model, fitted on 30 months of weekly sales and spend. For each channel it reports orders per $1,000 of spend at current levels, with 95% intervals:

Paid search: 3.2 (2.1 to 4.3), on $50,000 a month
Paid social: 1.1 (−0.4 to 2.6), on $80,000 a month

Each order brings $400 of contribution. The analyst's notes add one line: in each of the last two years, the social and search budgets were raised together in November and cut together in January, and search has never run above $60,000 a month. The CMO proposes moving all $80,000 from social to search next quarter: "social doesn't work, and search returns three times as much."

The question

Do you approve the move? Say what you do with the $80,000 next quarter instead, and what result would change your answer.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

No. Break-even is $1,000 ÷ $400 = 2.5 orders per $1,000, and each row has to be read against both zero and that line.

Social's interval runs from −0.4 to 2.6. It crosses zero, so the model cannot say social works, and it reaches past 2.5, so the model cannot say social fails. The analyst's note explains why: social and search rose and fell together, so the model cannot split their effects. "Social doesn't work" is not in the table.

Search's 3.2 sits above break-even, but its interval dips to 2.1, below it. And 3.2 is the average return at $50,000 a month; moving $80,000 in takes search to $130,000, more than twice anything in the history, so the model is extrapolating. If the extra dollars return half the average, 1.6, the $80,000 buys 128 orders. Social's $80,000 may be buying anywhere from none to 2.6 × 80 = 208. The move could gain 128 orders or lose 80.

Next quarter, keep social's $80,000 in place and add $15,000 a month to search in half the regions only, so search's return above the $60,000 the history covers is measured against the other half. Run a geo test on social, dark in a quarter of the regions for six weeks plus two weeks of carryover, and feed its result into the model as a calibration point.

What would change the answer: a geo test putting social under about 1.5 orders per $1,000, with an interval wholly below 2.5. Then cut social, in steps, toward whichever channel the calibrated model says buys most at the margin.

Rubric

Computes break-even, 2.5 orders per $1,000, from the $400 contribution
Reads social's interval as "cannot tell", not "does not work", and ties it to the budgets that moved together
Notes that search's interval dips below break-even and that 3.2 is an average at $50,000, not the return on $80,000 more
Puts a number on the range of outcomes of the CMO's move
Proposes an experiment on social with a size and a duration, and says how its result goes back into the model
Names the result that would justify the cut

Where answers go wrong

"Approve it: 3.2 is nearly three times 1.1." It compares two point estimates as if they were measurements, and ignores both intervals and the fact that the next dollar in search returns less than the average.

"The model is useless; both intervals are too wide." Search's interval clears zero, which is real information, and the cure for social's interval is an experiment that calibrates the model, not discarding it.

"Split the difference: move $40,000." Half of an unsupported move is still unsupported. It has no number behind it and no test that would show whether it was right.