The business question

A B2B software company scores every account for churn risk each quarter. The model ranks accounts well, as the AUC below shows. The open question is the one nobody’s dashboard answers: where do you draw the line?

To place the cutoff on your own model score given what each error costs you, try the free threshold optimizer.

Above the line, a customer success manager reaches out with a retention offer. Below it, nothing happens. Move the line down and you spend discounts on accounts that were never going to leave. Move it up and you lose accounts you could have saved.

The instinct is to pick the cutoff that classifies the most accounts correctly. That instinct is expensive, and this worked example measures exactly how expensive, because the two mistakes do not cost the same money:

Accuracy treats those as interchangeable. Finance does not.

The data

One row per account, as a churn-scoring pipeline would emit it. These accounts are synthetic. The code below simulates 1,200 of them with a fixed random seed. The risk score is built from the same hidden risk that decides who churns, so it ranks well by design. The dollar figures show how the method works; they are not a forecast for any real company.

column meaning
account_id the account
segment SMB / Mid-Market / Enterprise
seats licensed seats
mrr_usd monthly recurring revenue, US dollars
churn_risk_score model output, 0 to 100, higher = more likely to churn
churned_next_quarter 1 = the account actually churned, 0 = it stayed
set.seed(20260824)

n <- 1200
segment <- sample(c("SMB", "Mid-Market", "Enterprise"), n, replace = TRUE,
                  prob = c(0.58, 0.30, 0.12))
seats <- ifelse(segment == "SMB",        sample(3:25,    n, replace = TRUE),
         ifelse(segment == "Mid-Market", sample(25:120,  n, replace = TRUE),
                                         sample(120:600, n, replace = TRUE)))
mrr_usd <- round(seats * runif(n, 28, 46), 2)

# latent risk -> actual churn; the score is a noisy read of the same latent risk
z          <- rnorm(n)
p_churn    <- plogis(-2.60 + 1.40 * z)
churned    <- rbinom(n, 1, p_churn)
score_raw  <- z + rnorm(n, 0, 0.35)
churn_risk_score <- round(100 * pnorm(score_raw))

accounts <- data.frame(
  account_id           = sprintf("ACC-%04d", seq_len(n)),
  segment              = segment,
  seats                = seats,
  mrr_usd              = mrr_usd,
  churn_risk_score     = churn_risk_score,
  churned_next_quarter = churned,
  stringsAsFactors     = FALSE
)
write.csv(accounts, "data.csv", row.names = FALSE)  # the copy shipped next to this page is roc_scores.csv

data.frame(accounts = n, churned = sum(churned),
           churn_rate_pct = round(100 * mean(churned), 2))
##   accounts churned churn_rate_pct
## 1     1200     161          13.42
kable(head(accounts, 8), caption = "First eight rows as the scoring pipeline emits them")
First eight rows as the scoring pipeline emits them
account_id segment seats mrr_usd churn_risk_score churned_next_quarter
ACC-0001 SMB 3 87.70 25 0
ACC-0002 Mid-Market 53 2045.00 91 0
ACC-0003 SMB 25 1094.24 69 0
ACC-0004 SMB 7 241.46 47 0
ACC-0005 Enterprise 572 23434.12 17 0
ACC-0006 SMB 4 165.49 46 0
ACC-0007 SMB 11 362.40 93 1
ACC-0008 Mid-Market 74 2635.77 44 0

Does the score work at all?

r <- roc(accounts$churned_next_quarter, accounts$churn_risk_score,
         quiet = TRUE, direction = "<")
auc_val <- as.numeric(auc(r))
auc_ci  <- as.numeric(ci.auc(r, method = "delong"))
c(AUC = round(auc_val, 4), ci_lo = round(auc_ci[1], 4), ci_hi = round(auc_ci[3], 4))
##    AUC  ci_lo  ci_hi 
## 0.8078 0.7730 0.8427

The AUC has a plain-English reading worth keeping: take one account that churned and one that did not, at random. The AUC is how often the churned one carries the higher risk score.

The cost model

Costs are per account and driven by its own revenue, so a mistake on an Enterprise account counts for more than the same mistake on a small one.

The save rate is an assumption, not a measurement, so it is named and tested below.

SAVE_RATE <- 0.60

total_cost <- function(threshold, save_rate = SAVE_RATE, d = accounts) {
  flag <- d$churn_risk_score >= threshold
  y    <- d$churned_next_quarter == 1
  m    <- d$mrr_usd
  sum(m[flag &  y]) * 2 + sum(m[flag &  y]) * (1 - save_rate) * 12 +   # TP
  sum(m[flag & !y]) * 2 +                                             # FP
  sum(m[!flag &  y]) * 12                                             # FN
}

confusion <- function(threshold, d = accounts) {
  flag <- d$churn_risk_score >= threshold
  y    <- d$churned_next_quarter == 1
  c(TP = sum(flag & y), FP = sum(flag & !y),
    FN = sum(!flag & y), TN = sum(!flag & !y))
}

Sweep every candidate cutoff

grid <- 0:100
sweep <- do.call(rbind, lapply(grid, function(th) {
  cm  <- confusion(th)
  sens <- cm["TP"] / (cm["TP"] + cm["FN"])
  spec <- cm["TN"] / (cm["TN"] + cm["FP"])
  data.frame(threshold = th, TP = cm["TP"], FP = cm["FP"], FN = cm["FN"], TN = cm["TN"],
             sensitivity = sens, specificity = spec,
             accuracy = (cm["TP"] + cm["TN"]) / nrow(accounts),
             youden = sens + spec - 1,
             cost = total_cost(th))
}))
rownames(sweep) <- NULL

th_acc    <- sweep$threshold[which.max(sweep$accuracy)]
th_youden <- sweep$threshold[which.max(sweep$youden)]
th_cost   <- sweep$threshold[which.min(sweep$cost)]
c(accuracy_max_at = th_acc, youden_max_at = th_youden, cost_min_at = th_cost)
## accuracy_max_at   youden_max_at     cost_min_at 
##              98              61              90
cands <- sweep[sweep$threshold %in% sort(unique(c(th_acc, th_youden, th_cost))), ]
cands$label <- ifelse(cands$threshold == th_acc,    "accuracy-max",
               ifelse(cands$threshold == th_youden, "Youden J (what the tool reports)",
                                                    "cost-min"))
kable(data.frame(
  cutoff      = cands$threshold,
  label       = cands$label,
  flagged     = cands$TP + cands$FP,
  TP = cands$TP, FP = cands$FP, FN = cands$FN, TN = cands$TN,
  sensitivity = round(cands$sensitivity, 4),
  specificity = round(cands$specificity, 4),
  accuracy    = round(cands$accuracy, 4),
  cost_usd    = round(cands$cost, 0)
), row.names = FALSE, caption = "Three defensible cutoffs, and what each one costs")
Three defensible cutoffs, and what each one costs
cutoff label flagged TP FP FN TN sensitivity specificity accuracy cost_usd
61 Youden J (what the tool reports) 451 128 323 33 716 0.7950 0.6891 0.7033 5354965
90 cost-min 139 66 73 95 966 0.4099 0.9297 0.8600 4781400
98 accuracy-max 29 21 8 140 1031 0.1304 0.9923 0.8767 5204217
cost_at_acc  <- sweep$cost[sweep$threshold == th_acc]
cost_at_best <- sweep$cost[sweep$threshold == th_cost]
c(cost_accuracy_max = round(cost_at_acc, 0),
  cost_cost_min     = round(cost_at_best, 0),
  penalty_usd       = round(cost_at_acc - cost_at_best, 0),
  penalty_pct       = round(100 * (cost_at_acc - cost_at_best) / cost_at_best, 2))
## cost_accuracy_max     cost_cost_min       penalty_usd       penalty_pct 
##        5204217.00        4781400.00         422816.00              8.84

Each cutoff is scored on the same 1,200 accounts that were used to find it, so the best cutoff here is a little flattering. On new accounts the best cutoff would move, and the cost curve is bumpy because only 161 accounts churned.

Is the answer robust to the save-rate assumption?

sens_tab <- do.call(rbind, lapply(c(0.40, 0.60, 0.80), function(s) {
  costs <- sapply(grid, total_cost, save_rate = s)
  data.frame(save_rate = s, cost_min_threshold = grid[which.min(costs)],
             cost_at_min = round(min(costs), 0),
             cost_at_accuracy_max = round(costs[grid == th_acc], 0),
             cost_at_youden = round(costs[grid == th_youden], 0),
             accuracy_penalty_pct = round(100 * (costs[grid == th_acc] - min(costs)) / min(costs), 2))
}))
kable(sens_tab, row.names = FALSE,
      caption = "The accuracy-max cutoff is never the cost-optimal one, at any save rate tested")
The accuracy-max cutoff is never the cost-optimal one, at any save rate tested
save_rate cost_min_threshold cost_at_min cost_at_accuracy_max cost_at_youden accuracy_penalty_pct
0.4 94 5219451 5350800 6190254 2.52
0.6 90 4781400 5204217 5354965 8.84
0.8 80 4207276 5057633 4519675 20.21

The charts a practitioner reads

plot(r, legacy.axes = TRUE, col = "#F97316", lwd = 2.6,
     xlab = "false positive rate (1 - specificity)", ylab = "true positive rate (sensitivity)",
     main = sprintf("ROC: churn risk score (AUC %.3f)", auc_val))
pts <- sweep[sweep$threshold %in% c(th_acc, th_youden, th_cost), ]
pt_col <- ifelse(pts$threshold == th_cost, "#d94141",
          ifelse(pts$threshold == th_youden, "#3fbf6f", "#5fa9dd"))
points(1 - pts$specificity, pts$sensitivity, pch = 19, cex = 1.5, col = pt_col)
text(1 - pts$specificity, pts$sensitivity,
     labels = sprintf(" %d", pts$threshold), adj = c(0, 1.6), cex = 0.9)
legend("bottomright", bty = "n", pch = 19,
       col = c("#d94141", "#3fbf6f", "#5fa9dd"),
       legend = c(sprintf("cost-min (%d)", th_cost),
                  sprintf("Youden J (%d)", th_youden),
                  sprintf("accuracy-max (%d)", th_acc)))

plot(sweep$threshold, sweep$cost / 1000, type = "l", lwd = 2.6, col = "#F97316",
     xlab = "cutoff (flag accounts scoring at or above)", ylab = "expected quarterly cost (US$ thousands)",
     main = "Cost by cutoff: the accurate cutoff sits on the expensive side of the curve")
abline(v = th_cost, col = "#d94141", lwd = 2, lty = 2)
abline(v = th_acc,  col = "#5fa9dd", lwd = 2, lty = 2)
text(th_cost, max(sweep$cost) / 1000, sprintf("cost-min %d", th_cost), pos = 4, col = "#d94141")
text(th_acc,  max(sweep$cost) / 1000 * 0.92, sprintf("accuracy-max %d", th_acc), pos = 4, col = "#5fa9dd")

What the lesson teaches

  1. AUC answers “does the score rank?”, not “where is the line?” A good AUC is a licence to choose a threshold, not a threshold.
  2. Accuracy silently assumes the two mistakes cost the same. They differ by 6x here (12 months of MRR against 2), and the accuracy-max cutoff costs more than the cost-min cutoff at every save rate tested.
  3. Both cost-blind rules cost more than the best cutoff, and they miss in opposite directions. Accuracy is highest when almost nobody is flagged (churners are the minority, so “do nothing” scores well), so it leaves 140 churners unflagged where the cost-min cutoff leaves 95. Youden’s J, the optimum the tool reports, over-corrects the other way and flags 451 accounts, most of which were never going to leave. Youden is a fair default when you have no cost model. Once you have one, use it.
  4. Write the cost model down. Two numbers (what a miss costs, what an offer costs) set what each cutoff costs, and the save rate alone moved the best cutoff from 80 to 94 here.
  5. Name the assumption and test it. The save rate is a guess. We tried 40%, 60% and 80%: the best cutoff moved from 80 to 94, and the accuracy-max cutoff cost more than the best one each time. Three values is a spot check, not a full range.
  6. The tool needs one row per account: the true outcome, and a numeric score where higher means more likely to churn.

What this does not tell you

This is one simulated set of accounts, so it cannot say what your best cutoff is. The 12x and 2x costs and the save rate are assumptions made up for the example. The cutoffs were chosen and scored on the same accounts, with no held-out check. The page also ignores things a real team would weigh: what a save is worth beyond one year, accounts that churn after the offer anyway, and the time a manager spends on each call.

Try it on your own data

Bring one row per account with the real outcome and your model’s score, set your own cost for a miss and for an offer, and compare the cutoffs side by side.

Your turn

Bring your own data and the question you actually need answered.

MCP AnalyticsData science by email Send us your data and your question, and we will reply. ds@mcpanalytics.ai