The Kolmogorov–Smirnov Test: Comparing Distributions
The Kolmogorov–Smirnov (KS) test answers a precise question: do two samples come from the same distribution (two-sample), or does one sample come from a specified reference distribution (one-sample)? Unlike PSI it needs no binning (it works directly on the empirical cumulative distribution function), which makes it the natural drift detector for continuous features.
Its central idea is geometric: line up the two CDFs and measure the single largest vertical gap between them. If that gap is bigger than sampling noise can plausibly explain, the distributions differ.
The empirical CDF
Given a sample , the empirical CDF (ECDF) is the step function that, at any point , reports the fraction of observations at or below :
It rises by at each observed value and is flat in between. By the Glivenko–Cantelli theorem, converges uniformly to the true CDF as , which is exactly what makes a CDF-gap statistic meaningful.
The D statistic
One-sample
Compare the ECDF against a fully specified reference CDF (e.g. ). The statistic is the supremum of the absolute gap:
In practice the supremum is attained at one of the data points, so it reduces to a finite maximum. Because the ECDF jumps at each , you must check the gap on both sides of every jump:
where are the order statistics.
Two-sample
Compare two ECDFs and directly — no reference distribution needed:
This is the version used in drift monitoring: is the training sample, is the production window.
Check both sides of every jump
Because the ECDF is a right-continuous step function, the largest gap can occur just before a jump, not at it. Evaluating only understates . The from-scratch implementation must take the max over both and at each point.
From D to a p-value
Under (same distribution), the scaled statistic converges to the Kolmogorov distribution, whose tail gives the p-value. The remarkable property: this limiting distribution is distribution-free. It does not depend on the underlying (for continuous data). The asymptotic tail is
For the two-sample test, replace with the effective size . A common large-sample rejection rule at level :
Worked example
Two small samples:
- ()
- ()
Walk left to right over all pooled points, tracking each ECDF (each step is ):
| 0.1 | 0.25 | 0.00 | 0.25 |
| 0.2 | 0.50 | 0.00 | 0.50 |
| 0.3 | 0.50 | 0.25 | 0.25 |
| 0.5 | 0.75 | 0.25 | 0.50 |
| 0.6 | 0.75 | 0.50 | 0.25 |
| 0.7 | 1.00 | 0.50 | 0.50 |
| 0.8 | 1.00 | 0.75 | 0.25 |
| 0.9 | 1.00 | 1.00 | 0.00 |
. The 5 % critical value is . Since , we fail to reject — with only 4 points each, this much separation is unremarkable. The same with would be wildly significant; KS power scales with sample size.
KS for drift detection
KS is a standard alternative to PSI for monitoring continuous features. The two answer subtly different questions:
| KS test | PSI | |
|---|---|---|
| Binning | None (works on ECDF) | Required (typically deciles) |
| Output | + a p-value | Unitless score, no p-value |
| Sensitivity | Largest single CDF gap: strongest near the distribution's center | Aggregate over all bins: spreads weight across the range |
| Failure with huge | Hypersensitive: flags tiny, irrelevant shifts as "significant" | Threshold is sample-size-stable (proportions, not counts) |
| Best for | Smaller windows; principled significance | Large production streams; fixed action thresholds |
At production scale, the p-value lies
With millions of rows, the KS test rejects for any difference, however trivial — statistical significance ≠ operational significance. Teams running KS on large streams either subsample to a fixed , or ignore the p-value and threshold on the effect size directly (e.g. alert at ), which is closer in spirit to how PSI thresholds are used.
Assumptions and limitations
- Continuous data. The distribution-free guarantee holds for continuous distributions. With ties (discrete or heavily rounded data) the standard p-value is conservative; use a permutation-based variant instead.
- Most sensitive near the center. tends to be largest where the CDF is steepest (the bulk), so KS is comparatively weak at detecting tail differences. If two distributions share a center but differ in the tails, KS may miss it.
- One-sample KS needs a fully specified . If you estimate 's parameters from the same data (e.g. fit then test normality), the standard critical values are wrong — use the Lilliefors correction for normality testing.
When the tails matter, use Anderson–Darling. The Anderson–Darling test weights the gap by , inflating sensitivity in the tails — the right choice when tail behavior is what you care about (e.g. risk, extreme values).
Related concepts
- Hypothesis Testing: Null, Alternative & p-values: the null-hypothesis framework the KS p-value lives inside
- Population Stability Index (PSI): the binned drift metric KS complements; same job, different trade-offs
- Bootstrap & Resampling: the resampling route to a KS p-value when ties break the distribution-free guarantee