Upload a CSV with a predicted probability and the true outcome, and get the reliability curve with confidence intervals, the Brier score decomposed into reliability, resolution and uncertainty, the calibration intercept and slope, and what Platt scaling or isotonic regression would buy. Free.
Free analyses run on up to 10,000 rows. Larger files are randomly sampled to that size — sign up to analyze your full dataset.
Binning the predictions and decomposing the Brier score...
Sent to — Reliability curve with binomial confidence intervals, the Brier score decomposed into reliability, resolution and uncertainty, calibration intercept and slope, expected and maximum calibration error, a binning-sensitivity check, Platt and isotonic recalibration, R code, and AI insights.
Analyze another fileRows are grouped into equal-width bins of predicted probability, and each bin's average prediction is compared against the share of its rows that actually were the positive outcome, with a 95% Wilson interval on that observed share. The row-weighted average gap is the expected calibration error and the largest gap is the maximum calibration error, reported both over bins holding enough rows to be evidence and over every bin. The Brier score is decomposed into reliability, resolution and uncertainty over those bins, with the leftover reported as a residual. A logistic regression of the outcome on the predicted logit gives the calibration intercept and slope (0 and 1 are ideal), and an offset version gives the calibration-in-the-large. Because the binning changes the numbers, the whole error calculation is repeated under several equal-width resolutions and an equal-count binning, and the report states whether the verdict held. Two remedies are fitted for comparison: Platt scaling, which is that same logistic recalibration applied back to the predictions, and isotonic regression, computed with the pool-adjacent-violators algorithm implemented directly. Discrimination is reported alongside as the rank AUC, together with the Brier score of a forecast that predicts the base rate for every row.
Use it when you have predicted probabilities and the outcomes they were predicting, and the number itself matters — because you feed it into an expected-value calculation, a price, a capital charge, or a threshold that someone will defend.
Not for asking how well a score ranks or where to set a cutoff (that is the ROC analysis), not for a raw score or logit that was never meant to be a probability, and not for multi-class outcomes beyond a one-against-the-rest reading.
Built for: Data scientists, risk analysts, actuaries, and operators who act on a model's probability rather than just its ranking
Typical data source: A table with one predicted probability per row and the true yes/no outcome that followed
Any table with a predicted probability and the outcome it was predicting. For example, a credit model's scored applications:
Minimum 30 rows · Best with 500-50,000 rows with at least 5 of each outcome
Standard-library analysis: are the predicted probabilities honest? Map a predicted probability and the true yes/no outcome and get the reliability curve (predicted versus observed rate per bin, against the perfect-calibration diagonal, with a binomial confidence interval on every bin), the Brier score decomposed into reliability, resolution and uncertainty, the calibration intercept and slope from a logistic recalibration, expected and maximum calibration error, the distribution of the predicted probabilities, a check that the verdict survives a different binning, and what Platt scaling and isotonic regression would buy. Answers the question 'when this model says 70%, does it happen 70% of the time?'
Observed outcome rate against predicted probability, one point per bin — the further a point sits from the diagonal, the further that part of the model is from meaning what it says.
The numbers behind the curve, with a binomial confidence interval on every observed rate so you can tell a real gap from a thin-bin accident.
The calibration intercept and slope, which separate 'too high overall' from 'too extreme at both ends', plus the typical and worst calibration errors.
The Brier score split into the part a recalibration could remove, the part that comes from the model knowing something, and the part fixed by the outcome itself.
Where the model's predictions actually sit — the regions that carry the weight in every calibration number above.
The same error recomputed under several binnings, so you can see whether the verdict is a property of the model or of the bin edges.
What Platt scaling and isotonic regression would buy, measured in-sample and therefore optimistic.
The ranking question alongside the honesty question, with the base-rate forecast as proof that they are not the same thing.
Plain-English interpretation — what the numbers mean, what's significant, and what to do next.
When the model says 70%, does it happen 70% of the time?
Map the predicted probability and the true outcome. You get the reliability curve with a confidence interval on every bin, the expected calibration error, and the calibration intercept and slope — which separate 'too high everywhere' from 'too extreme at both ends', because those two faults need different fixes.
See our FAQ for details on pricing, data privacy, and how the analysis works. Every report includes a Methodology section showing the statistical test, assumptions checked, and diagnostics run.
Run any analysis on your own data — validated R analyses, interactive reports, AI insights, and PDF export.
Try Free — No Credit CardTell us what went wrong, in your own words. We capture the page you're on automatically, so no need to describe where you are.