Platt Scaling and Isotonic RegressionOpen in Colab

Platt Scaling and Isotonic Regression

Both are post-hoc calibration maps: given a trained classifier that outputs a score s(x)s(x) (a logit, an SVM margin, a random-forest vote fraction), they learn a function gg on a held-out set so that g(s(x))g(s(x)) behaves like a real probability. Neither touches the model's weights, and (because both are monotonic) neither changes the ranking of examples, so ROC-AUC and top-1 accuracy are untouched. They differ in the shape of map they can express and in how much calibration data they need.

The setup

Train the model on the training fold. Then, on a separate calibration fold, collect pairs (si,yi)(s_i, y_i) — the model's score and the true label. Fitting gg on the calibration fold (never the training fold) is the whole game: fit it on data the model already memorized and you learn the model's overconfidence rather than correct it.

Both methods require gg to be non-decreasing — a higher score should never map to a lower probability. That monotonicity is what preserves ranking; the methods differ in the family gg is drawn from.

Platt scaling: a logistic fit on the scores

Platt's method (originally for SVM outputs) assumes the distortion is sigmoidal and fits a two-parameter logistic map:

g(s)=P(y=1∣s)=11+exp⁡(A s+B)g(s) = P(y = 1 \mid s) = \frac{1}{1 + \exp(A\,s + B)}

AA and BB are chosen by minimizing log loss on the calibration set. Because AA controls the slope and BB the horizontal shift, this is exactly a 1-D logistic regression with ss as the single feature. With a well-ranked score A<0A < 0 (probability rises with score); ∣A∣|A| sets how sharply, BB recenters the 50% point.

The regularized-target trick. Fitting to raw 0/10/1 labels overfits badly on small calibration sets and drives probabilities to the extremes — the very problem we're fixing. Platt instead fits to smoothed targets derived from a Bayesian (Beta) prior on the class counts N+N_+ and N−N_-:

ti={N++1N++2yi=11N−+2yi=0t_i = \begin{cases} \dfrac{N_+ + 1}{N_+ + 2} & y_i = 1 \\[2mm] \dfrac{1}{N_- + 2} & y_i = 0 \end{cases}

A positive is asked to hit N++1N++2\tfrac{N_+ + 1}{N_+ + 2} (a hair under 1), a negative 1N−+2\tfrac{1}{N_- + 2} (a hair over 0). This built-in label smoothing is why Platt scaling stays stable with only a few hundred calibration points.

When it fits well: the true reliability curve is close to sigmoidal — a good match for max-margin models (SVMs) and for boosted models, whose scores are pushed toward the extremes in a characteristically sigmoidal way. When it fails: the miscalibration has a non-sigmoidal shape (e.g. an S the wrong way, or a kink); two parameters simply can't bend to it, leaving residual error.

Isotonic regression: the non-parametric monotone fit

Isotonic regression drops the sigmoid assumption. It finds the best non-decreasing step function (any monotone shape) by minimizing squared error subject to a monotonicity constraint:

min⁡g non-decreasing∑i(g(si)−yi)2\min_{g \text{ non-decreasing}} \sum_i \big(g(s_i) - y_i\big)^2

The exact solution is computed by the Pool-Adjacent-Violators Algorithm (PAV), O(n)O(n) after sorting:

  1. Sort the calibration pairs by score ss. Start with each label yiy_i as its own block, value =yi= y_i.
  2. Scan left to right. Whenever a block's value exceeds the next block's (a monotonicity violation), pool the two blocks into one whose value is their (size-weighted) mean.
  3. Pooling can create a new violation with the block on the left, so back up and re-pool until the whole sequence is non-decreasing.
  4. The fitted probability for a score is its final block's mean; interpolate between block boundaries for unseen scores.

Worked PAV trace. Calibration labels sorted by ascending score: [ 0, 1, 0, 0, 1, 1 ][\,0,\ 1,\ 0,\ 0,\ 1,\ 1\,].

StepBlocks (value × size)Action
Init0 · 1 · 0 · 0 · 1 · 1—
10, 1, …0≤10 \le 1 ok, advance
20, 0.5(×2), 0, 1, 11>01 > 0 → pool {1,0} → mean 0.50.5
30, 0.33(×3), 1, 10.5>00.5 > 0 → pool {1,0,0} → mean 0.3‾0.\overline{3}
Done0, 0.33, 0.33, 0.33, 1, 1fully non-decreasing

The five distinct labels collapse to a monotone staircase [0, 0.33, 0.33, 0.33, 1, 1][0,\ 0.33,\ 0.33,\ 0.33,\ 1,\ 1] — the unique least-squares monotone fit.

Step through that pooling, then watch the two calibrators compete on the same scores:

Pool-Adjacent-Violators — merging your way to the exact monotone fit

1/10

Six calibration points, already sorted by model score, with labels 0, 1, 0, 0, 1, 1. Isotonic regression wants the best non-decreasing step function through them — best in the least-squares sense — and PAV finds it exactly, in one pass, by only ever fixing local violations.

python
1def pav(y):
2 # every label starts as its own block
3 blocks = [[v, 1] for v in y] # value, size
4 i = 0
5 while i < len(blocks) - 1:
6 if blocks[i][0] <= blocks[i + 1][0]:
7 i += 1 # no violation
8 continue
9 # violation: pool into the weighted mean
10 v0, n0 = blocks[i]
11 v1, n1 = blocks[i + 1]
12 pooled = (v0 * n0 + v1 * n1) / (n0 + n1)
13 blocks[i:i + 2] = [[pooled, n0 + n1]]
14 i = max(i - 1, 0) # back up
15 return blocks
state
blocks (value × size)
0×1
1×1
0×1
0×1
1×1
1×1
start: every label its own block
calibration points, sorted by score →fitted p

The labels are 0/1 but the fit is not: pooling produces fractional values, and those fractions are exactly the calibrated probabilities.

PAV on the six labels worked above: scan left to right, and whenever a block exceeds its neighbour, pool the pair into their weighted mean and back up — because pooling can create a new violation behind you. Every step is a purely local merge, yet the staircase it lands on is the exact least-squares monotone fit. The final step pits isotonic against Platt scaling on a deliberately non-sigmoidal miscalibration, and both halves of the usual advice show up: Platt wins on small calibration sets, isotonic wins once there is enough data, and Platt's error floor never improves however much data it gets.

What to notice as you step:

  • Every move is local, but the answer is global. PAV only ever compares adjacent blocks and only ever merges — it never looks ahead and never moves a block down. Yet the staircase it lands on is the exact least-squares monotone fit, which is what makes it O(n)O(n) after the sort.
  • Backing up is not an optimization, it is required. Pooling {1, 0} into 0.5 immediately creates the possibility of a new violation to its left, and at the next step exactly that happens. Skip the back-up and the output is not monotone.
  • Pooling is the algorithm admitting ignorance. Three points collapse to a single block at 13\tfrac13 — the data cannot justify separating them, so it averages rather than inventing structure.
  • The last step is the practical choice. On a miscalibration a sigmoid cannot express (three flat shelves), Platt wins at n=50n = 50 where isotonic is fitting noise across ~17 blocks; isotonic pulls ahead by n=500n = 500 and nearly reaches the oracle floor at n=1500n = 1500. Meanwhile Platt improves by almost nothing across a 30× increase in data — that residual is bias, not variance, and no amount of calibration data buys it off.

When it wins: enough calibration data (rule of thumb ≳1000\gtrsim 1000 points) and a miscalibration shape a sigmoid can't capture. When it fails: small calibration sets — with too few points it overfits, producing a jagged staircase and, because it is piecewise-constant, sharp jumps and flat plateaus. It also can't extrapolate beyond the score range it saw.

Platt vs. isotonic vs. temperature scaling

Temperature scalingPlatt scalingIsotonic regression
Map familysoftmax(z/T)\text{softmax}(z / T)logistic 1/(1+eAs+B)1/(1+e^{As+B})any monotone step function
Free parameters1 (TT)2 (A,BA, B)non-parametric (O(n)O(n) blocks)
Assumptionuniform logit stretchsigmoidal distortionmonotonicity only
Data neededvery littlelittle (hundreds)more (≳1000\gtrsim 1000)
Multi-classnative (one TT)one-vs-rest per classone-vs-rest per class
Failure modecan't fix non-uniform miscalibrationcan't fix non-sigmoidal shapeoverfits on small data; jagged/flat
Changes ranking?NoNoNo

Rule of thumb: temperature scaling for deep nets (one knob, tiny validation set); Platt for SVMs / boosted models and small calibration sets; isotonic when you have plenty of calibration data and the reliability curve is oddly shaped. All three are monotonic, so none costs you accuracy or AUC.

Common pitfalls

  1. Calibrating on the training set. The model already fits it; you'll learn its overconfidence instead of correcting it. Always use a held-out calibration fold (or cross-validated out-of-fold predictions, as sklearn's CalibratedClassifierCV does).
  2. Isotonic on too little data. Below ~1000 calibration points it overfits into a jagged staircase; prefer Platt there.
  3. Assuming calibration transfers under shift. A map fit on one distribution silently decays when the deployment distribution moves — recalibrate on target-domain data.
  4. Forgetting these only rescale, never re-rank. If the underlying scores rank examples badly (low AUC), no monotone map can fix that — calibration and discrimination are separate problems.
  5. Calibrating an already-good model into worse shape. On a well-calibrated model, a data-hungry isotonic fit can add error; check ECE before and after and keep the map only if it helps.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Platt Scaling and Isotonic Regres…ROC Curve and AUC: Deriva…Calibration & UncertaintyClassification & Regressi…