AI in the operation · Scoring and measurement

Act on the churn score or not

Decide whether a churn score earns a retention action, which action, and what holdout result would change the call, with each break-even worked.

casemediumExercise · 3 min · 50 of 60

On this page

A churn score says who is likely to leave. The action it triggers is paid for on everyone it flags, including the customers who were staying anyway.

The situation

A subscription product, smaller than module 7's invoicing app but with the course's default customer, has 2,000 customers paying $50 a month, $40 of it margin, at 5% monthly churn: 100 cancel each month, and a customer is worth $800 of expected margin. An analytics vendor's churn score ranks every customer monthly; on last quarter's data its top decile, 200 customers, held 50 of a month's 100 cancellations, 25% precision against a 5% base rate.

The retention lead proposes giving the top 200 three months at half price: $25 a month off, $75 of margin each. The success team could instead call each of them: 20 minutes, priced at the course rep's loaded $75 an hour, $25 a call. Nobody has measured how many at-risk customers either action keeps; the vendor's deck says "up to 40%". Assume a saved customer is worth the full $800, since the save buys the customer's whole remaining life, and size any test with n ≈ 16 × p(1 − p) ÷ δ² per arm, at 5% significance and 80% power.

The question

Do you act on the score, with which action and on whom? Say what result would change your decision.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The score finds who is likely to leave, not who can be kept, and either action is paid on all 200, of whom 150 were staying.

The discount costs 200 × $75 = $15,000 a month, $11,250 of it to customers who would have stayed. It breaks even at 15,000 ÷ 800 ≈ 19 saves of the 50, a 38% save rate: the top of the vendor's "up to 40%". The calls cost 200 × $25 = $5,000 and break even at 5,000 ÷ 800 = 6.25 saves, 12.5%.

Act, with calls, on the top decile. The call's bar is a third of the discount's, and if it saves nobody it loses $5,000 a month, not $15,000. On the call, ask why they are leaving, and offer the discount only to those who name price, so it goes to the people for whom price is the reason.

Measure with a holdout, knowing its limits. Confirming the call's break-even, a fall from 25% to about 21.9% churn, needs 16 × 0.25 × 0.75 ÷ 0.031² ≈ 3,100 customers per arm, over two years of flagged customers, so that bar is decided on the cost of being wrong rather than proven. Hold out a random 50 of each month's 200 for three months: 150 against 450 can detect a gap of about 11 points, close to what the discount would need (38% of 25% is 9.5 points).

What changes the decision: a gap of 10 points or more says retention works at a scale where testing the price offer more widely pays; no visible gap and leavers citing a missing feature say the money belongs in the product.

Rubric

Counts the 150 customers in the top decile who were staying and receive the action anyway
Computes break-even saves for both actions: about 19 (38%) for the discount, about 6.25 (12.5%) for the call
Chooses one action and one population, and states what it costs if it saves no one
Designs a holdout and states which effect it can detect and which it cannot
Names the result that would reverse the decision, in the units the holdout measures

Where answers go wrong

"25% precision means the score is wrong three times in four; ignore it." A random 200 would hold 10 of the month's churners; the score's 200 hold 50, five times as many. The score is usable. The question is what the action costs per save, which this answer never asks. The lesson on predictive scoring reads precision against the base rate.

"Give the discount; the vendor says up to 40% are saved." At exactly 40% the discount returns 20 × $800 = $16,000 on $15,000, a margin of $1,000 resting on the vendor's best case, while $11,250 a month goes to customers who were not leaving.

"Run an A/B test until it is significant, then decide." With 200 flagged customers a month, a 3-point effect takes over two years to confirm, so the test either runs indefinitely or gets read early. A test plan states the effect it can see before it starts.