Predictive scoring
Read a lead or churn score against its base rate, then set the call threshold from sales capacity and a call's break-even instead of the model's default.
On this page
A lead score ranks leads; it does not say how many to call. The threshold is set by two numbers outside the model: how many calls the team can make, and what a call costs against what a converted lead is worth. Every score conversation starts from the base rate, because a score is worth exactly its lift over calling at random, and a vendor's headline accuracy usually hides it.
Base rates first
The running month: 5,000 new leads, of whom 4% convert, so 200 converters. A rep who calls 300 leads at random finds 300 × 4% = 12 of them.
The number vendors quote is accuracy, and at a 4% base rate it is the wrong number. A model that predicts "will not convert" for every lead is right about 4,800 of 5,000, 96% accurate, and finds nobody. Two numbers say what a score is worth instead:
- Precision at a cut: of the leads above the cut, the share that convert.
- Recall at a cut: of all the month's converters, the share above the cut.
A score with 20% precision in its top decile finds 100 converters in 500 leads: five times the base rate (20% ÷ 4%), and half of all 200 converters (recall of 50%) in a tenth of the list.
Reading a gains table
Ask for the table on your own last quarter, not on the vendor's benchmark: rank last quarter's leads by the score they had at the time, cut them into ten equal groups, and count who converted in an average month.
| Decile | Leads | Converters | Precision | Recall so far |
|---|---|---|---|---|
| 1 | 500 | 100 | 20% | 50% |
| 2 | 500 | 40 | 8% | 70% |
| 3 | 500 | 20 | 4% | 80% |
| 4 | 500 | 12 | 2.4% | 86% |
| 5 | 500 | 8 | 1.6% | 90% |
| 6 to 10 | 2,500 | 20 | 0.8% | 100% |
The table also checks calibration. Many scores claim to be probabilities: a lead scored 20 should convert about 20% of the time. Bucket last quarter's leads by score and compare each bucket's conversion rate with its score. A score that says 20 and delivers 9 still ranks, but every break-even sum built on its numbers is wrong by a factor of two.
Capacity sets the threshold
Two cuts compete, and the threshold is whichever binds first.
The break-even cut. A call costs a rep's time. At the course's $150,000 a year loaded, a rep costs $75 an hour over about 2,000 working hours: the course's 2,112 a year, 176 a month, less about 14 days of leave. A call with its research and logging takes 20 minutes: $25. Only the cost is borrowed: these reps qualify inbound leads, not close deals as in PLG versus sales-led. A converted lead is worth the course's default customer, $800 of expected margin. A call pays when precision × $800 ≥ $25, so when precision is at least 25 ÷ 800 = 3.1%. Deciles 1 to 3 clear it; decile 4, at 2.4%, does not. On value alone, the team should call 1,500 leads a month. The $800 is lifetime margin, which answers whether a call ever pays; on 12-month margin, $368, the bar is 6.8% and decile 3 drops out.
The capacity cut. The team makes 300 calls a month. That is fewer than the top decile's 500, so capacity binds long before break-even does.
The threshold therefore comes from capacity, not from the model. Suppose the top decile is scored more finely: 100 leads at 34, 200 at 21, 200 at 12, which between them hold the decile's 100 converters (34 + 42 + 24). The 300 calls take the 34s and the 21s and expect 34 + 42 = 76 converters, 25% precision, against 12 for 300 random calls. That sentence is the score's value to the sales team.
The gap between the two cuts is the decision the model hands back: 1,200 leads worth a call and nobody to make it. The next 200 calls, the leads at 12, would find about 24 converters at $96 of expected margin per $25 call. That is the case for a second rep, priced.
Ties: a threshold is one number
A CRM routes leads with a rule such as "score ≥ 21". If the score comes in coarse buckets, the cut cannot fall inside one. Raise capacity to 400 and the next band is 200 leads at 12: calling all of them exceeds 400, and the rule cannot pick 100 of the 200 without some other ordering, such as arrival time, which nobody chose on purpose. So the threshold stays at 21 and 100 calls are spare. Report the spare calls rather than hide them; the team can spend them by hand, or ask the vendor for a finer score. A score delivered only as deciles, all 500 of the top decile tied at one value, would leave a 300-call team with no usable cut at all. Pick the score threshold writes this rule as a function.
Found is not caused
The arithmetic above credits every converter a rep reaches to the call. Some would have converted anyway, most of all at the top of the ranking, where the score has learned to spot leads who already booked a demo. What a call is worth is the conversions it causes. A holdout settles it, slowly: leave a random 10% of the called band uncalled and compare. At 30 held-out leads a month, telling 25% from 19% takes over a year, about 3,800 leads with 380 held out, so the holdout runs continuously. If it shows the called leads at 25% and the held-out ones at 19%, the call causes 6 points, worth 6% × $800 = $48 against a $25 call: still worth making, but a quarter of the $200 the found count suggested. The design of that holdout is the subject of experiments, holdouts and power.
The same caution matters more for churn scores. In a business smaller than module 7's invoicing app, with 2,000 customers at the default 5% monthly churn, 100 leave each month; a churn score whose top decile of 200 holds 50 of them has 25% precision, the same shape as the lead table. But a retention action costs money on all 200, including the 150 who were staying, and the customers most certain to leave may be the least persuadable. Act on the churn score or not works that case.
Where the score runs
Scores are rebuilt as a batch, usually nightly, from CRM fields and product events. That is fast enough for a call list made each morning; streaming is worth its cost only when a rep must call within minutes of a sign-up, and batch versus stream gives the staleness arithmetic. Whatever the cadence, store each lead's score as it was when the list was made. A gains table built from today's scores on last quarter's leads flatters the model, because today's score has seen what happened since.
In practice
To sales: "We call the top 300 by score. In an average month last quarter, that band held 76 of the 200 converters, against 12 for 300 random calls." To the vendor: "Show us the gains table on our last quarter, with scores as they were at the time, and the calibration by bucket." Accuracy is not an answer to either.
The mistake that costs money is cutting at the score's default label instead of at capacity. If "hot" means 900 leads and the team can call 300, reps choose among the 900 by instinct and the ranking is wasted; if "hot" means 120, the other 180 calls go to leads picked at random. Either way the team pays for a score and calls as if it had none.
Check yourself
1. A second vendor's score has 12% precision in its top decile on your 5,000 leads, against the current score's 20%. How many converters does each find in your 300 calls?
Assuming precision is flat inside the decile, the second score finds 300 × 12% = 36 and the current one 300 × 20% = 60; the finer-grained current score, which puts the 34s and 21s first, finds 76. Both beat 12 at random. Like for like, the second finds 24 fewer converters a month (60 − 36), $19,200 of expected margin, so it would have to cost that much less to be worth switching to.
2. A second rep doubles capacity to 600 calls. Where does the threshold go, and what do the extra 300 calls find?
The 34s, 21s and 12s are 500 leads; the next band is 500 leads at 8, which does not fit whole, so the threshold is 12 and 100 calls are spare. The 200 extra calls that fit find about 24 converters, $19,200 of expected margin a month, against a rep at $12,500 a month. Hire, if a holdout shows enough of the 24 are caused by the call, and ask the vendor to split the 8s so the spare 100 calls are not wasted.
3. A holdout run long enough to tell 3 points apart shows the top 300 convert at 25% when called and 22% when not. Do you keep calling them?
The call causes 3 points: 3% × $800 = $24 per $25 call, just below break-even. Before stopping, check the lower band: if the leads at 12 convert at 12% called and 6% uncalled, a call there causes 6 points, $48, and the calls should move down the list. The best leads by score are not always the best leads to call.