Calibration Curves: Is Your Model's 70% Really 70%?

Your fraud detection model flagged 200 transactions with "70% probability of fraud." You investigated all 200. Only 82 were actually fraudulent. That's 41%, not 70%. Your model discriminates well—it catches more fraud than random chance—but the probabilities are wildly miscalibrated. When you tell a customer "70% risk," you need that number to mean something. This is where calibration curves come in, and here's the step-by-step methodology to verify your model's probabilities match reality.

Step 1: Understand What You're Actually Measuring

Before we check calibration, let's be clear about what we're testing. Model calibration answers one question: when my model predicts X% probability, does the event actually occur X% of the time?

This is different from discrimination, which asks "can my model rank-order cases correctly?" A model can have excellent discrimination (AUC = 0.90) but terrible calibration. That fraud model above? It probably has good AUC—it does identify risky transactions. But when it says "70%," it means "these are higher-risk than average," not "70% of these will be fraud."

Most machine learning models optimize for discrimination. Random forests, gradient boosted trees, and neural networks all try to maximize predictive accuracy. The probabilities they output are often poorly calibrated. That's not a bug—it's a consequence of their training objective.

Step 2: Set Up Your Calibration Test (The Right Way)

Here's the experimental design for a proper calibration check:

What You Need

• Predicted probabilities: Your model's output for each case (between 0 and 1)
• Actual outcomes: What actually happened (0 or 1, failure or success)
• Held-out test data: Cases the model has never seen before (critical—calibration on training data is meaningless)
• Minimum sample size: At least 500 cases, ideally 1,000+ for stable estimates

The Methodology

The calibration curve uses a simple binning approach:

1. Group predictions into bins: Sort all predictions into 10 bins (0-10%, 10-20%, ..., 90-100%)
2. Calculate observed frequency in each bin: What proportion of cases in each bin actually had the event?
3. Plot predicted vs. observed: X-axis = predicted probability, Y-axis = observed frequency
4. Compare to the diagonal: Perfect calibration = points fall on the 45-degree line

Let's work through a real example.

Step 3: Run the Analysis (Worked Example)

You've built a logistic regression model to predict customer churn. You have 2,000 customers in your test set with predicted churn probabilities and actual 90-day outcomes. Here's how to check calibration systematically.

Prepare Your Data

Your CSV needs exactly two columns:

predicted_probability,actual_outcome
0.23,0
0.67,1
0.45,0
0.89,1
...

The predicted_probability is your model's output (0 to 1). The actual_outcome is what happened (0 = no churn, 1 = churned).

Calculate Binned Calibration

Using 10 bins, we group predictions and calculate observed rates:

Predicted Range	Mean Prediction	Observed Rate	N Cases	Calibration Gap
0-10%	5.2%	4.8%	180	-0.4%
10-20%	15.1%	14.2%	210	-0.9%
20-30%	24.8%	23.1%	195	-1.7%
30-40%	34.5%	31.8%	220	-2.7%
40-50%	44.9%	48.3%	198	+3.4%
50-60%	55.2%	58.7%	186	+3.5%
60-70%	64.8%	69.4%	201	+4.6%
70-80%	74.3%	81.2%	215	+6.9%
80-90%	84.7%	90.1%	223	+5.4%
90-100%	94.2%	96.8%	172	+2.6%

This table tells a story. Low-risk predictions (0-40% range) are slightly overestimating churn—observed rates are lower than predicted. High-risk predictions (60-90% range) are underestimating—more customers churned than predicted.

The calibration gap averages +2.1 percentage points. That means when you predict 70% churn risk, the actual rate is closer to 77%. Not catastrophic, but meaningful if you're using these probabilities to prioritize retention offers.

Calculate Brier Score

The Brier score is the mean squared error of probability predictions:

Brier Score = (1/N) × Σ(predicted_prob - actual_outcome)²

For this churn model: Brier Score = 0.184

Is that good? Compare to the baseline: predicting the overall churn rate (32%) for everyone.

Baseline Brier = (1/N) × Σ(0.32 - actual_outcome)²
Baseline Brier = 0.218

Your model (0.184) beats the baseline (0.218). That's a 15.6% improvement in Brier score. The model adds value, but there's room to improve calibration.

Step 4: Interpret the Calibration Curve Pattern

The shape of your calibration curve tells you what's wrong (and how to fix it). Here are the four patterns I see constantly:

Pattern 1: Below the Diagonal (Overconfident Model)

Your curve sits consistently below the 45-degree line. Predicted probabilities are too high. When you predict 60%, reality is 45%.

Common culprits: Random forests, gradient boosted trees, neural networks trained with heavy regularization. These models push probabilities toward extremes.

Fix: Apply Platt scaling or isotonic regression to the raw probabilities. This recalibrates without changing the rank order.

Pattern 2: Above the Diagonal (Underconfident Model)

Your curve sits above the line. Predicted probabilities are too conservative. When you predict 40%, reality is 55%.

Common culprits: Logistic regression with strong L2 regularization, naive Bayes, models trained on imbalanced data with class weights.

Fix: Reduce regularization strength, or recalibrate with Platt scaling. For class imbalance, adjust the decision threshold instead of warping probabilities.

Pattern 3: S-Shaped Curve (Sigmoid Miscalibration)

Low predictions are too high, middle range is okay, high predictions are too low (or vice versa). The curve bends away from the diagonal at both ends.

Common culprits: Support vector machines with RBF kernels, models with poorly tuned probability thresholds, models trained on different distributions than deployment.

Fix: Platt scaling was literally invented for this problem (fixing SVM probabilities). It fits a logistic curve to your calibration plot.

Pattern 4: Noisy but Centered (Well-Calibrated Model)

Points scatter around the diagonal with no systematic pattern. Some bins are above, some below, but the average gap is near zero.

Common culprits: Logistic regression with minimal regularization, calibrated boosted trees, well-tuned neural networks, properly scaled ensembles.

Fix: You're already well-calibrated. The noise is sampling variability. Don't over-correct.

Step 5: Decision Points—When Does Calibration Actually Matter?

Not every model needs perfect calibration. Here's when you can skip this step, and when it's non-negotiable:

Calibration is Critical When:

• You use probabilities directly in decisions: Dynamic pricing, risk-adjusted insurance premiums, resource allocation based on predicted need
• You show probabilities to users: "Your fraud risk score is 68%" needs to mean 68%, not "higher than average"
• You're combining multiple models: Ensembles weight predictions by probability—miscalibration propagates through the stack
• Cost-benefit analysis depends on probabilities: Expected value calculations require accurate probabilities, not just rank order
• You're reporting to regulators: Financial services, healthcare, and credit models often require calibration documentation

Calibration Matters Less When:

• You only use a fixed threshold: Classify as positive if predicted probability > 0.5. Calibration doesn't change classifications (though it helps explain them)
• You're only rank-ordering: "Show me the top 100 riskiest cases" doesn't require calibrated probabilities, just correct ranking
• You're doing exploratory analysis: Finding patterns and generating hypotheses doesn't require perfect probabilities
• The model is a feature in another model: If these predictions feed into a meta-model, let the meta-model handle calibration

Before spending time on calibration, ask: how will I use these probabilities? If the answer is "apply a threshold and classify," you can probably skip it. If the answer involves the actual numeric probability, calibrate first.

Step 6: Fix Miscalibration (Three Methods That Work)

You've identified miscalibration. Now what? Here are three recalibration methods, ranked by when to use them:

Method 1: Platt Scaling (Start Here)

Platt scaling fits a logistic regression model to map your raw probabilities to calibrated probabilities:

calibrated_prob = 1 / (1 + exp(-(A × raw_prob + B)))

You fit A and B on a held-out calibration set (separate from training and test data). This corrects sigmoid-shaped miscalibration.

When to use: First choice for most models. Works well with 500+ calibration samples. Particularly effective for SVMs, random forests, and gradient boosting.

Limitation: Assumes a parametric sigmoid relationship. If your calibration curve is more complex (e.g., wavy), isotonic regression works better.

Method 2: Isotonic Regression (For Complex Patterns)

Isotonic regression fits a non-parametric step function that's constrained to be monotonically increasing. It can match any calibration curve shape.

When to use: When Platt scaling doesn't fix the problem. When your calibration curve has multiple inflection points. When you have 2,000+ calibration samples (it needs more data than Platt scaling).

Limitation: Can overfit with small samples. Doesn't extrapolate well beyond the range of calibration data.

Method 3: Beta Calibration (For Extreme Probabilities)

Beta calibration extends Platt scaling with three parameters instead of two. It handles cases where probabilities cluster near 0 or 1 (common with highly confident models).

When to use: When your model outputs many probabilities near 0.01 or 0.99. When Platt scaling compresses the probability range too much.

Limitation: Requires even more calibration data than isotonic regression (3,000+ samples recommended).

The Data Split for Calibration

Critical point: you need three separate datasets for proper calibration:

1. Training set (60%): Train your model
2. Calibration set (20%): Fit the calibration function (Platt, isotonic, or beta)
3. Test set (20%): Evaluate final calibrated model performance

If you calibrate on the same data you test on, you'll get overly optimistic calibration metrics. If you calibrate on training data, you're calibrating to the fit, not the generalization.

Common Mistakes That Invalidate Your Calibration Analysis

I've reviewed hundreds of calibration analyses. Here are the mistakes that make results meaningless:

Mistake 1: Testing Calibration on Training Data

Your model has seen the training data. Of course it looks calibrated there. The test is whether it generalizes to new cases. Always use held-out test data. Always.

Mistake 2: Using Too Many Bins with Small Samples

You have 800 test cases and create 20 bins. That's 40 cases per bin. The observed frequency in each bin is noisy, and your calibration curve looks terrible even if the model is fine.

Rule: Aim for 50+ cases per bin. With 1,000 samples, use 10 bins. With 500 samples, use 5 bins. With 10,000 samples, you can use 20 bins.

Mistake 3: Ignoring Class Imbalance

Your dataset is 95% negative, 5% positive. You create 10 bins. The first 8 bins have almost no positive cases—observed frequencies are either 0% or 100% based on 1-2 examples.

Fix: Use quantile-based binning instead of equal-width bins. This ensures each bin has similar sample sizes. Or focus on reliability diagrams that account for bin sizes.

Mistake 4: Confusing Calibration with Discrimination

You see a poor calibration curve and conclude your model is useless. But the AUC is 0.85. The model discriminates well—it just needs recalibration. Don't throw it out; fix the probabilities.

Mistake 5: Over-Interpreting Small Calibration Gaps

One bin shows predicted 45%, observed 41%. You panic and retrain the model. But with 80 cases in that bin, the 95% confidence interval is ±11 percentage points. That "gap" is statistical noise.

Rule: Only worry about gaps larger than 2 × sqrt(p(1-p)/n), where p is the predicted probability and n is bin size. For p=0.45 and n=80, that's ±11pp. Your 4pp gap is not significant.

Mistake 6: Calibrating Without a Separate Calibration Set

You fit Platt scaling parameters on your test set, then report test set calibration. You've overfit the calibration and your results are too optimistic.

Fix: Use a three-way split: train (60%), calibrate (20%), test (20%). Only the final test set should be used to report performance.

What Your Calibration Report Should Include

When you run calibration analysis, here's what to look at (and what to report to stakeholders):

Essential Outputs

1. Calibration plot: Predicted probability (X) vs. observed frequency (Y), with 45-degree reference line
2. Brier score: Overall calibration quality (lower is better)
3. Brier skill score: Improvement over baseline (% reduction in Brier score)
4. Bin-level table: For each bin, show predicted range, mean prediction, observed rate, sample size, and calibration gap
5. Confidence intervals: 95% CIs around observed frequencies in each bin (to distinguish signal from noise)

Advanced Diagnostics

1. Expected Calibration Error (ECE): Weighted average of absolute calibration gaps across bins
2. Maximum Calibration Error (MCE): Largest absolute gap in any bin (worst-case calibration)
3. Reliability diagram: Calibration plot with bin sizes shown (larger dots = more samples)
4. Hosmer-Lemeshow test: Formal statistical test of calibration (though less useful than visual inspection)

Most stakeholders don't need all of this. Show them the calibration plot, explain the Brier score, and highlight any bins with large gaps. That's enough to make decisions.

Real-World Application: When We Found a 23% Calibration Gap

A lending platform used a gradient boosted tree model to predict default risk. The model had AUC = 0.78, which seemed reasonable. They used predicted probabilities to set interest rates: higher predicted default risk = higher rate to compensate for risk.

We ran calibration analysis on their most recent 5,000 loans. The results were alarming:

• For loans predicted at 20% default risk, observed default rate was 8%
• For loans predicted at 60% default risk, observed default rate was 37%
• Average calibration gap: 14 percentage points
• Worst bin: predicted 75%, observed 52% (23pp gap)

The model was systematically overestimating default risk. They were charging borrowers interest rates based on inflated risk estimates. Regulatory problem. Customer fairness problem. Revenue problem (they were losing good borrowers to competitors with better pricing).

The Fix

We applied Platt scaling using a held-out calibration set of 2,000 loans. Post-calibration results:

• Average calibration gap: 2.1 percentage points
• Brier score improved from 0.142 to 0.118
• All bins within ±5pp of perfect calibration
• AUC unchanged at 0.78 (calibration doesn't affect discrimination)

They recalibrated the model, adjusted interest rates based on corrected probabilities, and submitted the updated methodology to their regulator. Six months later, analysis of loan performance data confirmed the calibrated probabilities matched observed default rates within expected confidence intervals.

Calibration for Different Model Types

Not all models miscalibrate the same way. Here's what to expect:

Logistic Regression

Default calibration: Excellent (if assumptions are met). Logistic regression directly optimizes log-loss, which encourages calibration.

When it breaks: Strong L2 regularization, class imbalance with sample weights, violated linearity assumptions.

Fix: Reduce regularization or recalibrate with Platt scaling. Check if you actually need regularization—you might be over-regularizing.

Random Forest

Default calibration: Poor. Random forests predict the proportion of trees voting for each class. This systematically underestimates extreme probabilities (they rarely predict below 10% or above 90%).

When it breaks: Always. Assume miscalibration and test it.

Fix: Platt scaling or isotonic regression. Both work well.

Gradient Boosted Trees (XGBoost, LightGBM, CatBoost)

Default calibration: Poor to moderate. These models optimize for ranking and often push probabilities toward extremes.

When it breaks: High learning rates, many boosting rounds, imbalanced classes.

Fix: Platt scaling. Some implementations (LightGBM) have built-in calibration options—use them.

Neural Networks

Default calibration: Highly variable. Depends on architecture, regularization, and training procedure. Modern networks (especially large ones) tend toward overconfidence.

When it breaks: Heavy dropout, batch normalization, insufficient training data, transfer learning without fine-tuning the final layer.

Fix: Temperature scaling (a simpler variant of Platt scaling) works well for neural networks. Or use isotonic regression.

Naive Bayes

Default calibration: Terrible. Naive Bayes assumes feature independence, which is almost never true. Probabilities are systematically biased toward extremes.

When it breaks: Always, but the ranking is often still useful.

Fix: Isotonic regression (works better than Platt scaling for Naive Bayes's highly non-linear miscalibration).

Support Vector Machines

Default calibration: Non-existent. SVMs don't output probabilities natively. The Platt scaling method was literally invented to generate probabilities from SVM decision scores.

When it breaks: By definition, unless you've already applied Platt scaling.

Fix: Use Platt scaling (it's the standard approach). Most SVM libraries include this as a built-in option.

Frequently Asked Questions

What's the difference between calibration and discrimination?

Discrimination measures whether your model can rank-order cases correctly (high-risk customers score higher than low-risk customers). Calibration measures whether your predicted probabilities match reality. A model can have excellent discrimination (AUC = 0.90) but terrible calibration (predicted 70% risk, observed 40% rate). Both matter. Discrimination tells you if the model is useful. Calibration tells you if the probabilities are trustworthy.

How many bins should I use for a calibration curve?

Start with 10 bins as the default. With fewer than 1,000 observations, use 5 bins. With more than 10,000 observations, you can use up to 20 bins. The goal is at least 30-50 observations per bin. Too few bins and you miss miscalibration patterns. Too many bins and you get noisy estimates with wide confidence intervals.

What's a good Brier score?

Brier scores range from 0 (perfect) to 1 (worst possible). For rare events (5% base rate), Brier < 0.05 is good. For balanced outcomes (50% base rate), Brier < 0.20 is reasonable. Compare your model's Brier score to the baseline: predicting the overall event rate for everyone. If your model's Brier score isn't meaningfully lower than baseline, the model isn't adding value.

Can I fix a poorly calibrated model?

Yes. Three common approaches: (1) Platt scaling fits a logistic regression to map raw scores to calibrated probabilities. Works well for SVMs and boosted trees. (2) Isotonic regression fits a non-parametric monotonic function. More flexible but requires more data. (3) Beta calibration extends Platt scaling for cases where probabilities cluster near 0 or 1. All three require a held-out calibration set separate from training and test data.

Should I calibrate before or after choosing a decision threshold?

Calibrate first, then choose your threshold. Calibration fixes the probabilities themselves. The threshold determines which predicted probability triggers action. If you're using the probabilities directly (e.g., showing customers their fraud risk score), calibration is essential. If you're only using the threshold (approve/deny decisions), calibration is less critical but still valuable for understanding model behavior and explaining decisions.

The Bottom Line: Trust But Verify

Your model says 70%. Before you make decisions based on that number—before you set prices, allocate resources, or show that probability to a customer—verify that 70% means 70%. The step-by-step methodology is straightforward: bin your predictions, calculate observed frequencies, plot the calibration curve, compute the Brier score. If you find miscalibration, fix it with Platt scaling or isotonic regression. If you don't check, you're making decisions based on numbers that might be systematically wrong.

Model calibration isn't optional for high-stakes predictions. When your fraud model flags transactions, when your churn model prioritizes retention offers, when your credit model sets interest rates—calibration determines whether those probabilities are trustworthy or just confident-sounding numbers.

Run the analysis. Check the curve. Fix what's broken. Then you can trust your 70%.

Want to verify your model's calibration right now? Upload your predictions and outcomes and get a complete calibration report with curve, Brier score, and bin-level diagnostics. The analysis runs in your browser—your data stays private. No signup required.