Your model scores every customer for churn. Where do you draw the line?

It depends on what a missed churner and a wasted offer cost you, and that is a business call, not a statistical one. If a missed churner costs 2, 5 or 10 times a wasted offer, the cheapest line on this data sits at a score of 0.269, 0.199 or 0.143. The usual default of 0.5 flags 94 of 946 customers and misses 80 of the 149 who churned.

The data are from the UCI Machine Learning Repository, an Iranian telecom churn sample: 3,150 customers, 495 of whom churned. Our model, a logistic regression fitted on 70% of the customers, gave each customer a score. We then scored the 30% it had not seen, 946 held-out customers, 149 of whom churned. Higher means more likely to churn. The line, or cutoff, flags a customer when the score is at or above it.

The costs are assumptions. Nothing in the data says what an offer costs or what a lost customer is worth, so we tried three ratios, always against a wasted offer as 1.

Our position: the line is a cost decision. Say what a miss and a wasted offer cost you, then draw it.

Population: 946 held-out customers, 149 of whom churned. A wasted offer costs 1 in every row. Cost per customer is the average over the 946. At the default 0.5, 94 customers are flagged: 69 churners caught, 25 false alarms, 80 churners missed and 772 customers correctly left alone.

The line is not exact. Across four other random splits of the same data, the cutoff for the 5-to-1 assumption ran from 0.140 to 0.243, against 0.199 on the split above.

Data: Iranian Churn [Dataset]. (2020). UCI Machine Learning Repository. https://doi.org/10.24432/C5JW3Z. CC BY 4.0. Changes: churn score by MCP Analytics (logistic regression fitted on 70% of customers, scored on the 30% held out); the cutoffs are computed on the held-out scores; the costs are stated assumptions, not data.

What this means for your decisions

• Do not accept 0.5 without checking what it costs you. At 0.5, 80 of the 149 churners are missed. If a missed churner costs 5 times a wasted offer, the cost per customer at 0.5 is 0.449, against 0.223 at the cheaper line.
• Move the line when the costs move. If a missed churner costs 2 times a wasted offer, the cheapest line is 0.269. If it costs 10, it is 0.143, and more customers are flagged, 286 instead of 206. A cheaper miss pushes the line up. A dearer one pulls it down.
• A range of ratios gives the same line. On this split, the 0.269 line is the cheapest if a missed churner costs between 1.36 and 2.80 times a wasted offer. On this split, the 0.199 line is the cheapest between 4.00 and 7.33, and the 0.143 line between 9.50 and 13.07.
• Two familiar rules land on the same lines. The cutoff with the best F1 is 0.269, the same as the 2-to-1 cost line. The cutoff from Youden's J is 0.199, the same as the 5-to-1 cost line. On this data they agree.
• These figures describe one split of one dataset. They do not show why customers churned, or that a save offer works, and the costs are assumptions you supply.

How to do it on your own file

You need two columns: an actual yes or no column (did the customer churn) and a score column, where higher means more likely to churn. Score rows the model did not see, and keep at least 10 of each class.

Our steps, for a first pass:

1. Pick the costs you can defend: what a wasted offer costs and what a missed churner costs.
2. Try the cutoffs one by one, and for each count the flagged customers, the false alarms and the misses.
3. Take the cutoff with the lowest total cost.
4. Rerun on a second random split to see how far the line moves.

The free threshold tool does the search on your file. Run the free threshold tool on your own scores. For background, see our practical guide to churn prediction and why 0.5 is not a default cutoff.

Questions readers ask

Why not just use 0.5?

Because 0.5 is a habit, not a cost decision. On this data it flags 94 of 946 customers and misses 80 of the 149 who churned. Whether that is the right trade depends on what a miss and a wasted offer cost you.

Where did the cost ratios come from?

We chose them. Nothing in the data says what an offer or a lost customer costs, so we tried three ratios: if a missed churner costs 2, 5 or 10 times a wasted offer. Replace them with yours.

How exact is the line?

Not very. Across four other random splits, the 5-to-1 cutoff ran from 0.140 to 0.243, against 0.199 on the split above. Treat any one cutoff as a place to start.

Does a lower line mean more customers are flagged for an offer?

Yes. At 0.269, 206 customers are flagged. At 0.143, 286 are.

Does this show that a save offer works?

No. It shows where the cost is lowest under the assumed costs. It does not say what an offer does to a customer.

How many customers do I need?

You need at least 10 of each class, churned and not, among the rows you score; that is the tool's minimum. Our reading: the fewer customers you have, the more the line moves.

Run it on your own file

The same question, on your own customers: set what a miss and a wasted offer cost you, and read the cheapest line. Run the free threshold tool on your own scores.