Decile Analysis: Gains, Lift, KS and Rank OrderingOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Decile Analysis: Gains, Lift, KS and Rank Ordering

A single AUC says the model ranks well. It does not say where the risk concentrated, how much of it you capture if you can only action the worst 10% of accounts, or whether the ordering breaks down somewhere in the middle. The decile table answers all three, which is why it is the first exhibit in a scorecard validation pack, a churn model review and a marketing campaign sizing deck.

The construction is mechanical: sort the population by score, cut it into equal-count bins, count the outcomes in each.

Building the table

  1. Score the validation population and sort descending by score, riskiest first.
  2. Cut into kk equal-count bins, usually k=10k = 10 (deciles). Equal count, not equal score width: each bin holds N/kN/k accounts whatever the score distribution looks like.
  3. Count outcomes per bin: bib_i bads and gig_i goods in bin ii.
  4. Accumulate from the top: Bi=∑j≤ibjB_i = \sum_{j \le i} b_j and Gi=∑j≤igjG_i = \sum_{j \le i} g_j.
  5. Derive the four columns below.
bad ratei=bini,lifti=bi/niBk/N,gainsi=BiBk,KSi=BiBk−GiGk\text{bad rate}_i = \frac{b_i}{n_i}, \qquad \text{lift}_i = \frac{b_i / n_i}{B_k / N}, \qquad \text{gains}_i = \frac{B_i}{B_k}, \qquad \text{KS}_i = \frac{B_i}{B_k} - \frac{G_i}{G_k}

Lift is the bin's bad rate divided by the overall bad rate: "accounts here are 3.7× as likely to go bad as a random account". Gains (also called the capture rate) is the cumulative share of all bads caught by targeting down to this bin. KS is the gap between the two cumulative curves, and its maximum over bins is the KS statistic.

A worked table

10,000 accounts, 500 bads, a 5.0% base rate.

Decilenbadsbad rateliftcum % badscum % goodsKScum lift
1100018518.5%3.7037.0%8.6%0.2843.70
2100011011.0%2.2059.0%17.9%0.4112.95
31000707.0%1.4073.0%27.7%0.4532.43
41000454.5%0.9082.0%37.8%0.4422.05
51000303.0%0.6088.0%48.0%0.4001.76
61000222.2%0.4492.4%58.3%0.3411.54
71000151.5%0.3095.4%68.7%0.2671.36
81000101.0%0.2097.4%79.1%0.1831.22
9100080.8%0.1699.0%89.5%0.0951.10
10100050.5%0.10100.0%100.0%0.0001.00

Reading it out loud, which is what the table is for:

  • Top decile lift is 3.70: the riskiest 10% of accounts default at 18.5%, against a 5.0% book average.
  • Capture at 20% is 59.0%: action the worst two deciles and you touch 59% of all the bads while reviewing a fifth of the book.
  • KS = 0.453 at decile 3: the point of maximum separation between bads and goods is the 30% cutoff, where 73.0% of bads and 27.7% of goods sit above the line.
  • Bad rate is strictly decreasing: no rank-order break. A decile riskier than the one above it means the score is not monotone in risk there, which is a finding, not a rounding error.

Where the deciles break, the cutoff usually is

The KS decile is a natural candidate cutoff because it maximises (bads caught − goods disturbed), but it is not automatically the right one. The right cutoff is where the marginal bad caught is worth the marginal good declined, which needs the cost ratio, not the KS. Use KS to summarise separation, and an expected-cost sweep to choose the threshold.

Play with it

Decile analysis: gains, lift and KS

cumulative % captured0255075100KS 0.535% of population targeted →bads capturedgoods capturedlift per bin (bad rate ÷ base rate)0×1×2×3×4×5×riskiest bin → safest bin
AUC (exact)0.856
Gini = 2·AUC−10.712
KS (exact)0.547
KS (10 bins)0.535
top-bin lift5.20×
binnbadsbad rateliftcum % badscum % goodsKS
150013026.0%5.20×52.0%7.8%0.442
2500479.4%1.88×70.8%17.3%0.535
3500224.4%0.88×79.6%27.4%0.522
4500214.2%0.84×88.0%37.5%0.505
5500153.0%0.60×94.0%47.7%0.463
650081.6%0.32×97.2%58.0%0.392
750040.8%0.16×98.8%68.5%0.303
850020.4%0.08×99.6%79.0%0.206
950010.2%0.04×100.0%89.5%0.105
1050000.0%0.00×100.0%100.0%0.000

Sample: 5,000 accounts, 250 bad (5.0% base rate). The KS row is highlighted. Binned KS (0.535) can only ever be ≤ the exact KS (0.547), because a bin grid finds the gap at its own cut points and nowhere else — push separation down until the bars stop descending cleanly and rose bins appear, the rank-order breaks that a validation pack has to explain.

Sort every account by score, cut into equal-sized bins, and count the bads in each. The gains curve is the cumulative share of bads captured (teal) against the cumulative share of goods (indigo); the dashed vertical bar is the KS statistic, the widest gap between them. Separation moves discrimination, prevalence moves only the base rate — watch AUC and KS hold while lift collapses, because lift is a ratio to the base rate and the other two are not.

Three things the viz makes testable:

  • Drag prevalence with separation fixed. AUC and KS barely move; top-decile lift collapses from 5.20× at a 5% base rate to 3.22× at 25%. Lift is a ratio to the base rate, and it carries a hard ceiling of 1/π1/\pi at prevalence π\pi: once a quarter of the book is bad, no bin can lift more than 4×, however good the ranking is. A "5× lift" claim with no prevalence attached means nothing.
  • Switch between 5, 10 and 20 bins. The binned KS is always at or below the exact KS computed over every distinct score, because a bin grid can only find the gap at its own cut points. More bins gets closer, at the cost of noisier per-bin rates.
  • Drag separation toward zero. Bins stop descending cleanly and rose rank-order breaks appear. On a real model, breaks concentrated in the middle deciles usually mean a feature is non-monotone in risk, and breaks in the bottom deciles usually mean too few bads down there to estimate a rate at all.

KS, and what it is not

The decile KS is the two-sample Kolmogorov–Smirnov statistic applied to the score distributions of bads and goods, evaluated on a grid of decile boundaries:

KS=max⁡t∣Fbad(t)−Fgood(t)∣\mathrm{KS} = \max_t \left| F_{\text{bad}}(t) - F_{\text{good}}(t) \right|

Two cautions travel with it. First, the binned KS understates the exact one, so a KS read off a decile table is not comparable to one computed on raw scores. Second, the KS test's p-value is not what a scorecard report means by "KS": nobody doubts that bads and goods have different score distributions, so the hypothesis test is vacuous here. The number is being used as a descriptive separation measure on a 0 to 1 scale, and typical banded readings are:

KSReading
below 0.20weak separation, hard to build a cutoff policy on
0.20 to 0.40typical for a live behavioural or application scorecard
0.40 to 0.60strong
above 0.70suspect leakage or a target that is partly an input

Gains, Gini and the decile approximation

The cumulative-gains curve plots cum % bads against cum % population; the ROC curve plots cum % bads against cum % goods. On this table the trapezoid area under the decile-grid ROC is 0.788, so the decile-approximated Gini is 2×0.788−1=0.5752 \times 0.788 - 1 = 0.575. That is an approximation from ten points, not the model's actual AUC. It is usually a little below the true pairwise c-statistic, because the grid draws chords across a concave ROC curve and a chord lies under the arc it spans. Report the exact c-statistic from raw scores and treat the table's implied Gini as a cross-check on the arithmetic, not as the headline number.

Deciles drift, so fix the cut points

Cutting deciles on today's score distribution and comparing them to last month's deciles compares two different populations, and a shift in the score distribution looks like a shift in risk. For monitoring, freeze the decile boundaries from the development sample and apply those fixed score cuts to every later period. Then a bin that swells or empties is a real population shift, measurable with PSI.

Hosmer–Lemeshow on the same bins

The table above measures discrimination: are the bads at the top? A model can get that right and still have the wrong level. The Hosmer–Lemeshow test reuses the same decile structure to check calibration, comparing observed bads per bin against the bads the model's own predicted probabilities expected:

C^=∑i=1g(Oi−Ei)2Ei(1−pˉi),Ei=nipˉi\hat{C} = \sum_{i=1}^{g} \frac{(O_i - E_i)^2}{E_i (1 - \bar{p}_i)}, \qquad E_i = n_i \bar{p}_i

where pˉi\bar{p}_i is the mean predicted probability in bin ii. Under the null (the model is calibrated) C^\hat{C} is approximately χ2\chi^2 with g−2g - 2 degrees of freedom when the bins come from a model fit on the same data.

Worked on the table above, with mean predicted probabilities 0.170, 0.115, 0.074, 0.048, 0.031, 0.021, 0.014, 0.010, 0.007, 0.004:

Decileobservedexpectedcontribution
1185170.01.595
2110115.00.246
37074.00.234
44548.00.197
53031.00.033
62221.00.049
71514.00.072
81010.00.000
987.00.144
1054.00.251

C^=2.82\hat{C} = 2.82 on 8 degrees of freedom, p=0.945p = 0.945: no evidence of miscalibration. Decile 1 supplies more than half the statistic, which is the useful part of the output. The test's verdict is much less informative than its per-bin residuals, and it has two well-known weaknesses: it is nearly powerless on small samples and rejects almost everything on large ones, and its value depends on the binning choice. Prefer a calibration curve plus ECE, and read Hosmer–Lemeshow as a bin-level residual table that happens to come with a p-value.

Common mistakes

  • Comparing lift across populations with different base rates. Lift is normalised by prevalence; a campaign list and a whole book will disagree even on an identical model.
  • Equal-width instead of equal-count bins. With a skewed score distribution, equal-width bins leave the top "decile" holding 40 accounts and 3 bads, and every rate in it is noise.
  • Reading a rank-order break at the bottom as a model defect. With 5 bads in decile 10, the bad rate's standard error is larger than the gap you are worrying about. Check the counts before the story.
  • Quoting a decile-table Gini as the model's AUC. It is a ten-point trapezoid approximation to it.
  • Re-cutting deciles every monitoring period and reporting the movement as drift in risk. Freeze the development cut points.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Decile Analysis: Gains, Lift, KS …Concordance: the c-statis…The Kolmogorov–Smirnov Te…Population Stability Inde…Uplift Modeling & Causal …Rank Ordering & Model Com…Calibration & Uncertainty