Platt Scaling and Isotonic Regression
Both are post-hoc calibration maps: given a trained classifier that outputs a score (a logit, an SVM margin, a random-forest vote fraction), they learn a function on a held-out set so that behaves like a real probability. Neither touches the model's weights, and (because both are monotonic) neither changes the ranking of examples, so ROC-AUC and top-1 accuracy are untouched. They differ in the shape of map they can express and in how much calibration data they need.
The setup
Train the model on the training fold. Then, on a separate calibration fold, collect pairs — the model's score and the true label. Fitting on the calibration fold (never the training fold) is the whole game: fit it on data the model already memorized and you learn the model's overconfidence rather than correct it.
Both methods require to be non-decreasing — a higher score should never map to a lower probability. That monotonicity is what preserves ranking; the methods differ in the family is drawn from.
Platt scaling: a logistic fit on the scores
Platt's method (originally for SVM outputs) assumes the distortion is sigmoidal and fits a two-parameter logistic map:
and are chosen by minimizing log loss on the calibration set. Because controls the slope and the horizontal shift, this is exactly a 1-D logistic regression with as the single feature. With a well-ranked score (probability rises with score); sets how sharply, recenters the 50% point.
The regularized-target trick. Fitting to raw labels overfits badly on small calibration sets and drives probabilities to the extremes — the very problem we're fixing. Platt instead fits to smoothed targets derived from a Bayesian (Beta) prior on the class counts and :
A positive is asked to hit (a hair under 1), a negative (a hair over 0). This built-in label smoothing is why Platt scaling stays stable with only a few hundred calibration points.
When it fits well: the true reliability curve is close to sigmoidal — a good match for max-margin models (SVMs) and for boosted models, whose scores are pushed toward the extremes in a characteristically sigmoidal way. When it fails: the miscalibration has a non-sigmoidal shape (e.g. an S the wrong way, or a kink); two parameters simply can't bend to it, leaving residual error.
Isotonic regression: the non-parametric monotone fit
Isotonic regression drops the sigmoid assumption. It finds the best non-decreasing step function (any monotone shape) by minimizing squared error subject to a monotonicity constraint:
The exact solution is computed by the Pool-Adjacent-Violators Algorithm (PAV), after sorting:
- Sort the calibration pairs by score . Start with each label as its own block, value .
- Scan left to right. Whenever a block's value exceeds the next block's (a monotonicity violation), pool the two blocks into one whose value is their (size-weighted) mean.
- Pooling can create a new violation with the block on the left, so back up and re-pool until the whole sequence is non-decreasing.
- The fitted probability for a score is its final block's mean; interpolate between block boundaries for unseen scores.
Worked PAV trace. Calibration labels sorted by ascending score: .
| Step | Blocks (value × size) | Action |
|---|---|---|
| Init | 0 · 1 · 0 · 0 · 1 · 1 | — |
| 1 | 0, 1, … | ok, advance |
| 2 | 0, 0.5(×2), 0, 1, 1 | → pool {1,0} → mean |
| 3 | 0, 0.33(×3), 1, 1 | → pool {1,0,0} → mean |
| Done | 0, 0.33, 0.33, 0.33, 1, 1 | fully non-decreasing |
The five distinct labels collapse to a monotone staircase — the unique least-squares monotone fit.
Step through that pooling, then watch the two calibrators compete on the same scores:
Pool-Adjacent-Violators — merging your way to the exact monotone fit
Six calibration points, already sorted by model score, with labels 0, 1, 0, 0, 1, 1. Isotonic regression wants the best non-decreasing step function through them — best in the least-squares sense — and PAV finds it exactly, in one pass, by only ever fixing local violations.
The labels are 0/1 but the fit is not: pooling produces fractional values, and those fractions are exactly the calibrated probabilities.
PAV on the six labels worked above: scan left to right, and whenever a block exceeds its neighbour, pool the pair into their weighted mean and back up — because pooling can create a new violation behind you. Every step is a purely local merge, yet the staircase it lands on is the exact least-squares monotone fit. The final step pits isotonic against Platt scaling on a deliberately non-sigmoidal miscalibration, and both halves of the usual advice show up: Platt wins on small calibration sets, isotonic wins once there is enough data, and Platt's error floor never improves however much data it gets.
What to notice as you step:
- Every move is local, but the answer is global. PAV only ever compares adjacent blocks and only ever merges — it never looks ahead and never moves a block down. Yet the staircase it lands on is the exact least-squares monotone fit, which is what makes it after the sort.
- Backing up is not an optimization, it is required. Pooling
{1, 0}into 0.5 immediately creates the possibility of a new violation to its left, and at the next step exactly that happens. Skip the back-up and the output is not monotone. - Pooling is the algorithm admitting ignorance. Three points collapse to a single block at — the data cannot justify separating them, so it averages rather than inventing structure.
- The last step is the practical choice. On a miscalibration a sigmoid cannot express (three flat shelves), Platt wins at where isotonic is fitting noise across ~17 blocks; isotonic pulls ahead by and nearly reaches the oracle floor at . Meanwhile Platt improves by almost nothing across a 30× increase in data — that residual is bias, not variance, and no amount of calibration data buys it off.
When it wins: enough calibration data (rule of thumb points) and a miscalibration shape a sigmoid can't capture. When it fails: small calibration sets — with too few points it overfits, producing a jagged staircase and, because it is piecewise-constant, sharp jumps and flat plateaus. It also can't extrapolate beyond the score range it saw.
Platt vs. isotonic vs. temperature scaling
| Temperature scaling | Platt scaling | Isotonic regression | |
|---|---|---|---|
| Map family | logistic | any monotone step function | |
| Free parameters | 1 () | 2 () | non-parametric ( blocks) |
| Assumption | uniform logit stretch | sigmoidal distortion | monotonicity only |
| Data needed | very little | little (hundreds) | more () |
| Multi-class | native (one ) | one-vs-rest per class | one-vs-rest per class |
| Failure mode | can't fix non-uniform miscalibration | can't fix non-sigmoidal shape | overfits on small data; jagged/flat |
| Changes ranking? | No | No | No |
Rule of thumb: temperature scaling for deep nets (one knob, tiny validation set); Platt for SVMs / boosted models and small calibration sets; isotonic when you have plenty of calibration data and the reliability curve is oddly shaped. All three are monotonic, so none costs you accuracy or AUC.
Common pitfalls
- Calibrating on the training set. The model already fits it; you'll learn its overconfidence instead of correcting it. Always use a held-out calibration fold (or cross-validated out-of-fold predictions, as
sklearn'sCalibratedClassifierCVdoes). - Isotonic on too little data. Below ~1000 calibration points it overfits into a jagged staircase; prefer Platt there.
- Assuming calibration transfers under shift. A map fit on one distribution silently decays when the deployment distribution moves — recalibrate on target-domain data.
- Forgetting these only rescale, never re-rank. If the underlying scores rank examples badly (low AUC), no monotone map can fix that — calibration and discrimination are separate problems.
- Calibrating an already-good model into worse shape. On a well-calibrated model, a data-hungry isotonic fit can add error; check ECE before and after and keep the map only if it helps.
Related concepts
- Calibration and Uncertainty: reliability diagrams, ECE, temperature scaling, and aleatoric vs. epistemic uncertainty
- Classification Metrics: the accuracy/precision metrics calibration complements
- ROC Curve and AUC: why a monotone calibration map leaves AUC unchanged