Population Stability Index (PSI): Measuring Distribution DriftOpen in Colab

Population Stability Index (PSI): Measuring Distribution Drift

The Population Stability Index (PSI) measures how much a feature's distribution has shifted between two samples — typically a reference (training) snapshot and a current (production) window. A single scalar per feature, computed daily or per-batch, is enough to catch upstream pipeline regressions, data-collection changes, and genuine covariate shift before they silently degrade model accuracy.

PSI originated in credit-risk modeling, where regulators required formal documentation that a scorecard's development population matched the one it was deployed on. It has since become the de facto standard for input drift monitoring in production ML systems across industries.

The formula

Divide the feature range into kk bins. Let pip_i be the reference (training) proportion in bin ii and qiq_i be the production proportion in bin ii. Then:

PSI=∑i=1k(pi−qi)⋅ln⁡ ⁣(piqi)\mathrm{PSI} = \sum_{i=1}^{k} (p_i - q_i) \cdot \ln\!\left(\frac{p_i}{q_i}\right)

Each term contributes the signed difference (pi−qi)(p_i - q_i) weighted by the log-ratio ln⁡(pi/qi)\ln(p_i/q_i). Because the difference and the log-ratio always share the same sign (when pi>qip_i > q_i both are positive, when pi<qip_i < q_i both are negative), every term is non-negative. PSI = 0 only when pi=qip_i = q_i for every bin. It is always finite and symmetric in pp and qq (proved below).

Threshold interpretation

PSIInterpretationRecommended action
<0.10< 0.10Stable: no significant shiftNo action; continue monitoring
0.100.10–0.250.25Moderate shiftInvestigate root cause; watch for one more window
≥0.25\geq 0.25Significant shiftRetrain (or recalibrate) and audit the upstream pipeline

These cut-points are rules of thumb — not statistical significance levels. A feature with high natural variability (e.g. daily traffic volume) may routinely hit 0.12 without any upstream problem; a highly stable feature (e.g. a customer's birth year) should raise an alarm at 0.05. Calibrate thresholds per feature using historical windows where the model was known to be healthy.

Binning strategies

Equal-frequency bins (recommended)

Sort the reference values and divide into kk quantile-based bins so each bin holds exactly 1/k1/k of the reference population. This gives pi=1/kp_i = 1/k for all ii by construction. The production proportions qiq_i are then computed by assigning production observations to the same fixed bin edges.

Why this is better than equal-width: Equal-width bins cut from min to max in fixed steps. A single outlier stretches one bin to hold almost all the reference mass, making the log-ratio in that bin large and volatile. Equal-frequency bins never leave a bin sparsely populated in the reference, so ln⁡(pi/qi)\ln(p_i/q_i) stays well-conditioned.

Rule of thumb: k=10k = 10 (deciles) is the standard default. Use k=20k = 20 for larger samples (> 10,000 rows); use k=5k = 5 for very small reference sets (< 500 rows) to keep each bin stable.

Equal-width bins

Suitable for categorical features or when you need bin edges to be human-readable (e.g. integer age ranges). For continuous features prefer equal-frequency.

Worked example: 10 bins, equal-frequency

Reference sample: 1,000 credit-score observations, binned at deciles. Production sample: 1,000 new observations scored against the same bin edges.

BinEdgeRef nnRef pip_iProd nnProd qiq_ipi−qip_i-q_iln⁡(pi/qi)\ln(p_i/q_i)Term
1≤ 5601000.1001200.120−0.020−0.1820.0036
2561–5901000.1001100.110−0.010−0.0950.0010
3591–6151000.1001050.105−0.005−0.0490.0002
4616–6351000.1001000.1000.0000.0000.0000
5636–6551000.100980.0980.0020.0200.0000
6656–6751000.100950.0950.0050.0510.0003
7676–6951000.100900.0900.0100.1050.0011
8696–7151000.100850.0850.0150.1630.0024
9716–7401000.100800.0800.0200.2230.0045
10> 7401000.1001170.117−0.017−0.1570.0027
Total1,0001.0001,0001.000PSI0.0158

Verdict: PSI ≈ 0.016 — well below 0.10. The credit-score distribution is stable.

Now suppose a data-pipeline bug introduced a 6-month lag in score updates, shifting the bulk of observations from bins 7–10 down to bins 1–3:

BinRef pip_iProd qiq_ipi−qip_i-q_iln⁡(pi/qi)\ln(p_i/q_i)Term
10.1000.250−0.150−0.9160.1374
20.1000.180−0.080−0.5880.0470
30.1000.130−0.030−0.2620.0079
4–60.3000.2800.0200.0690.0014
70.1000.0600.0400.5110.0204
80.1000.0400.0600.9160.0550
90.1000.0300.0701.2040.0843
100.1000.0300.0701.2040.0843
PSI0.4377

Verdict: PSI ≈ 0.44 — far above 0.25. The distribution has shifted dramatically toward lower scores; immediate investigation and likely retraining are warranted.

Largest terms reveal where the shift happened

Sort the per-bin terms in descending order to identify where in the distribution the drift is concentrated. In the second example, bins 1 and 2 account for 0.184 of the 0.438 total — the low end of the score range drove almost half the PSI. That diagnostic points directly at "something suppressed recent high scores."

Edge case: zero proportions

If any bin has pi=0p_i = 0 or qi=0q_i = 0, the term ln⁡(pi/qi)\ln(p_i/q_i) is undefined (log of zero or division by zero), and PSI blows up.

Standard fix: replace zero proportions with a small floor ε\varepsilon before computing:

p~i=max⁡(pi, ε),q~i=max⁡(qi, ε)\tilde{p}_i = \max(p_i,\, \varepsilon), \quad \tilde{q}_i = \max(q_i,\, \varepsilon)

then renormalize so ∑ip~i=1\sum_i \tilde{p}_i = 1 and ∑iq~i=1\sum_i \tilde{q}_i = 1. A common default is ε=10−4\varepsilon = 10^{-4}; another is ε=1/(2n)\varepsilon = 1/(2n) where nn is the sample size.

A zero production bin is itself diagnostic: a region of the feature space that existed at training time has completely disappeared in production. Flag these bins separately in your dashboard — they often indicate a schema change, a category being deprecated, or a pipeline filtering error.

Why PSI equals the Jeffreys divergence

PSI is the symmetrized KL divergence (Jeffreys divergence) between the reference and production distributions:

PSI=DKL(P ∥ Q)+DKL(Q ∥ P)\mathrm{PSI} = D_{\mathrm{KL}}(P \,\|\, Q) + D_{\mathrm{KL}}(Q \,\|\, P)

To verify: DKL(P ∥ Q)=∑ipiln⁡(pi/qi)D_{\mathrm{KL}}(P \,\|\, Q) = \sum_i p_i \ln(p_i/q_i) and DKL(Q ∥ P)=∑iqiln⁡(qi/pi)=−∑iqiln⁡(pi/qi)D_{\mathrm{KL}}(Q \,\|\, P) = \sum_i q_i \ln(q_i/p_i) = -\sum_i q_i \ln(p_i/q_i). Summing:

DKL(P ∥ Q)+DKL(Q ∥ P)=∑i(pi−qi)ln⁡ ⁣(piqi)=PSID_{\mathrm{KL}}(P \,\|\, Q) + D_{\mathrm{KL}}(Q \,\|\, P) = \sum_i (p_i - q_i)\ln\!\left(\frac{p_i}{q_i}\right) = \mathrm{PSI}

This explains two properties:

  • Symmetry: swapping reference and production gives the same PSI value. There is no privileged direction.
  • Non-negativity: both KL terms are ≥ 0 by Gibbs' inequality, so PSI ≥ 0 always.

KL divergence alone is directional. DKL(P ∥ Q)D_{\mathrm{KL}}(P \,\|\, Q) measures the extra nats needed to encode samples drawn from PP using a code optimized for QQ. Using bare KL as a drift metric forces you to choose which distribution is "reference" in the log-ratio. PSI avoids that choice by treating both symmetrically.

Categorical features

PSI applies directly to categorical features. Set each level to be its own bin. The formula is identical:

PSI=∑ℓ∈L(pℓ−qℓ)ln⁡ ⁣(pℓqqℓ)\mathrm{PSI} = \sum_{\ell \in \mathcal{L}} (p_\ell - q_\ell)\ln\!\left(\frac{p_\ell}{\vphantom{q}q_\ell}\right)

Grouping rare levels: with 50+ categories, levels that appear in < 1–5 % of the reference sample will have unstable per-bin log-ratios. A standard practice is to group everything below a relative-frequency threshold into an "Other" bucket before computing PSI.

New levels in production: a category that appears in production but not in training gets pℓ=0p_\ell = 0, which requires the ε\varepsilon-floor fix above. A new level is also worth flagging separately — it may represent a product category that didn't exist at training time.

What PSI does not capture

PSI is a marginal metric: it operates on one feature at a time. Three blind spots:

Joint distributional shift. If age and income were independent in training but become positively correlated in production, per-feature PSIs can both stay near zero while the joint distribution has shifted significantly. Options for joint drift: Maximum Mean Discrepancy (MMD), a two-sample permutation test on the raw feature matrix, or a classifier-based test (train a binary model to distinguish training from production; AUROC > 0.6 suggests detectable drift).

Ordering within bins. Equal-frequency bins suppress information about the shape of shifts within a bin. A distribution that bimodally concentrates at the edges of a bin can have the same qiq_i as a unimodal one centered in the middle of that bin.

Outcome drift. PSI measures covariate shift — p(x)p(x) changing. Label shift (p(y)p(y) changing) and concept drift (p(y∣x)p(y|x) changing) are not captured. Track prediction-score distributions and, when labels arrive, rolling accuracy metrics separately.

Actionable response guide

PSI bandFirst stepsDecision
<0.10< 0.10NoneContinue normal monitoring
0.100.10–0.250.25Identify which bins shifted; cross-check with pipeline changelogs and upstream data sources; compare with neighboring features that share the same pipelineWait one more window; retrain if PSI persists or climbs
≥0.25\geq 0.25Determine drift type (covariate vs concept via prediction-distribution check); correlate timing with deploys or world events; inspect zero bins separatelyTrigger retrain pipeline; recalibrate output confidence scores; page the data-platform team if cause is a pipeline bug

Correlate across features. Five related features spiking together almost always means a pipeline bug or upstream schema change — one root cause explains all. A single feature drifting in isolation more often reflects a real-world shift in that variable.

Debounce. Require two to three consecutive windows above threshold before triggering a retrain. A single spiked window is frequently a transient (a backfill, an API returning nulls for an hour). Debouncing cuts false-alarm retrains without significantly delaying the response to genuine drift.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Population Stability Index (PSI):…Decile Analysis: Gains, L…The Kolmogorov–Smirnov Te…Rank Ordering & Model Com…Continual Learning & Test…Monitoring & Observability