The Kolmogorov–Smirnov Test: Comparing DistributionsOpen in Colab

The Kolmogorov–Smirnov Test: Comparing Distributions

The Kolmogorov–Smirnov (KS) test answers a precise question: do two samples come from the same distribution (two-sample), or does one sample come from a specified reference distribution (one-sample)? Unlike PSI it needs no binning (it works directly on the empirical cumulative distribution function), which makes it the natural drift detector for continuous features.

Its central idea is geometric: line up the two CDFs and measure the single largest vertical gap between them. If that gap is bigger than sampling noise can plausibly explain, the distributions differ.

The empirical CDF

Given a sample x1,…,xnx_1, \ldots, x_n, the empirical CDF (ECDF) is the step function that, at any point xx, reports the fraction of observations at or below xx:

Fn(x)=1n∑i=1n1[xi≤x]F_n(x) = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}[x_i \le x]

It rises by 1/n1/n at each observed value and is flat in between. By the Glivenko–Cantelli theorem, FnF_n converges uniformly to the true CDF FF as n→∞n \to \infty, which is exactly what makes a CDF-gap statistic meaningful.

The D statistic

One-sample

Compare the ECDF FnF_n against a fully specified reference CDF F0F_0 (e.g. N(0,1)\mathcal{N}(0,1)). The statistic is the supremum of the absolute gap:

Dn=sup⁡x ∣Fn(x)−F0(x)∣D_n = \sup_x \, \bigl| F_n(x) - F_0(x) \bigr|

In practice the supremum is attained at one of the data points, so it reduces to a finite maximum. Because the ECDF jumps at each x(i)x_{(i)}, you must check the gap on both sides of every jump:

Dn=max⁡1≤i≤nmax⁡ ⁣(∣in−F0(x(i))∣,  ∣i−1n−F0(x(i))∣)D_n = \max_{1 \le i \le n} \max\!\left( \left|\tfrac{i}{n} - F_0(x_{(i)})\right|,\; \left|\tfrac{i-1}{n} - F_0(x_{(i)})\right| \right)

where x(1)≤⋯≤x(n)x_{(1)} \le \cdots \le x_{(n)} are the order statistics.

Two-sample

Compare two ECDFs FnF_n and GmG_m directly — no reference distribution needed:

Dn,m=sup⁡x ∣Fn(x)−Gm(x)∣D_{n,m} = \sup_x \, \bigl| F_n(x) - G_m(x) \bigr|

This is the version used in drift monitoring: FnF_n is the training sample, GmG_m is the production window.

Check both sides of every jump

Because the ECDF is a right-continuous step function, the largest gap can occur just before a jump, not at it. Evaluating only ∣Fn(x(i))−F0(x(i))∣|F_n(x_{(i)}) - F_0(x_{(i)})| understates DD. The from-scratch implementation must take the max over both i/ni/n and (i−1)/n(i-1)/n at each point.

From D to a p-value

Under H0H_0 (same distribution), the scaled statistic converges to the Kolmogorov distribution, whose tail gives the p-value. The remarkable property: this limiting distribution is distribution-free. It does not depend on the underlying FF (for continuous data). The asymptotic tail is

P ⁣(n Dn>t)→2∑k=1∞(−1)k−1e−2k2t2P\!\left(\sqrt{n}\, D_n > t\right) \to 2 \sum_{k=1}^{\infty} (-1)^{k-1} e^{-2 k^2 t^2}

For the two-sample test, replace n\sqrt{n} with the effective size nmn+m\sqrt{\tfrac{nm}{n+m}}. A common large-sample rejection rule at level α\alpha:

Dn,m>c(α)n+mnm,c(0.05)≈1.36,c(0.01)≈1.63D_{n,m} > c(\alpha)\sqrt{\frac{n+m}{nm}}, \qquad c(0.05) \approx 1.36,\quad c(0.01) \approx 1.63

Worked example

Two small samples:

  • A={0.1, 0.2, 0.5, 0.7}A = \{0.1,\ 0.2,\ 0.5,\ 0.7\} (n=4n = 4)
  • B={0.3, 0.6, 0.8, 0.9}B = \{0.3,\ 0.6,\ 0.8,\ 0.9\} (m=4m = 4)

Walk left to right over all pooled points, tracking each ECDF (each step is 1/4=0.251/4 = 0.25):

xxFA(x)F_A(x)GB(x)G_B(x)∣FA−GB∣\lvert F_A - G_B\rvert
0.10.250.000.25
0.20.500.000.50
0.30.500.250.25
0.50.750.250.50
0.60.750.500.25
0.71.000.500.50
0.81.000.750.25
0.91.001.000.00

D4,4=0.50D_{4,4} = 0.50. The 5 % critical value is 1.36(4+4)/(4⋅4)=1.360.5≈0.9621.36\sqrt{(4+4)/(4\cdot 4)} = 1.36\sqrt{0.5} \approx 0.962. Since 0.50<0.9620.50 < 0.962, we fail to reject H0H_0 — with only 4 points each, this much separation is unremarkable. The same D=0.5D = 0.5 with n=m=200n = m = 200 would be wildly significant; KS power scales with sample size.

KS for drift detection

KS is a standard alternative to PSI for monitoring continuous features. The two answer subtly different questions:

KS testPSI
BinningNone (works on ECDF)Required (typically deciles)
OutputDD + a p-valueUnitless score, no p-value
SensitivityLargest single CDF gap: strongest near the distribution's centerAggregate over all bins: spreads weight across the range
Failure with huge nnHypersensitive: flags tiny, irrelevant shifts as "significant"Threshold is sample-size-stable (proportions, not counts)
Best forSmaller windows; principled significanceLarge production streams; fixed action thresholds

At production scale, the p-value lies

With millions of rows, the KS test rejects H0H_0 for any difference, however trivial — statistical significance ≠ operational significance. Teams running KS on large streams either subsample to a fixed nn, or ignore the p-value and threshold on the effect size DD directly (e.g. alert at D>0.1D > 0.1), which is closer in spirit to how PSI thresholds are used.

Assumptions and limitations

  • Continuous data. The distribution-free guarantee holds for continuous distributions. With ties (discrete or heavily rounded data) the standard p-value is conservative; use a permutation-based variant instead.
  • Most sensitive near the center. DD tends to be largest where the CDF is steepest (the bulk), so KS is comparatively weak at detecting tail differences. If two distributions share a center but differ in the tails, KS may miss it.
  • One-sample KS needs a fully specified F0F_0. If you estimate F0F_0's parameters from the same data (e.g. fit then test normality), the standard critical values are wrong — use the Lilliefors correction for normality testing.

When the tails matter, use Anderson–Darling. The Anderson–Darling test weights the gap by 1/F0(x)(1−F0(x))1/\sqrt{F_0(x)(1-F_0(x))}, inflating sensitivity in the tails — the right choice when tail behavior is what you care about (e.g. risk, extreme values).

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
The Kolmogorov–Smirnov Test: Comp…Decile Analysis: Gains, L…The Bootstrap: Resampling…Hypothesis Testing: Null,…Population Stability Inde…Hypothesis Testing & p-va…Monitoring & Observability