Rank Ordering & Model Comparison
This lesson stays in the evaluation slot of the ML project loop, and it fixes two specific things that were breaking in the earlier lessons.
Lesson 1 measured a classifier at one threshold: precision, recall, F1 all assume you have already chosen a cutoff. Most production scoring problems have not chosen one yet, and some never will (a collections queue works top-down until the team runs out of hours). Ranking quality has to be measurable before the cutoff exists.
And every metric so far has been a point estimate with no error bar. "Model A: 0.86 AUC, model B: 0.84" is two numbers from one finite test set. Whether that gap survives another sample is a question none of the metrics so far can answer, and it is the question a model review actually asks.
Concordance: quality of ranking, no threshold required
Take one positive and one negative. Does the model score the positive higher? Count over every such pair and you have the whole story:
- concordant (): the positive scored higher, the model got this pair right
- discordant (): the negative scored higher, a ranking error
- tied (): equal scores, no preference expressed
On eight loan applicants (3 defaults, 5 repaid) the 15 pairs come out , , , and four familiar statistics fall straight out of those counts:
The c-statistic is the AUC, computed without drawing a curve or picking a threshold: it is the probability that a random positive outranks a random negative. Somers' D is , and the Gini coefficient a credit scorecard reports is the same number again. Three names, one quantity.
The fourth, Goodman–Kruskal gamma, divides by and deletes tied pairs. On a continuous score that changes nothing. On a coarse score it flatters the model badly, which is the trap the wiki page below makes concrete: bucket the same eight applicants into three risk grades and falls from 0.767 to 0.700 while gamma rises from 0.571 to 0.600.
Concordance is invariant to monotone transforms
Squash the scores through a sigmoid, rescale to 0 to 1000, add 7: the pair counts do not move, so does not move. That invariance is exactly why a high AUC tells you nothing about whether the predicted probabilities are right. Discrimination and calibration are two separate audits.
Exercise
A model scored 5 positives and 5 negatives. Of the 25 positive-negative pairs, 18 are concordant, 5 are discordant and 2 are tied. What is the c-statistic?
Need a hint?
The c-statistic scores a tied pair as half a win, and keeps it in the denominator.
Where did the risk actually land?
says the ranking is decent. It does not say what you get if you can only action the worst 10% of accounts, which is the question the business is asking. The decile table answers it: sort by score, cut into ten equal-count bins, count outcomes per bin.
Four columns do the work:
| Column | Definition | Reads as |
|---|---|---|
| bad rate | bads ÷ accounts in bin | how risky this slice is |
| lift | bin bad rate ÷ overall bad rate | how much better than random targeting |
| gains (capture) | cumulative % of all bads down to this bin | what you catch by actioning this far |
| KS | cum % bads − cum % goods | separation, maximised at the best cutoff bin |
Decile analysis: gains, lift and KS
| bin | n | bads | bad rate | lift | cum % bads | cum % goods | KS |
|---|---|---|---|---|---|---|---|
| 1 | 500 | 130 | 26.0% | 5.20× | 52.0% | 7.8% | 0.442 |
| 2 | 500 | 47 | 9.4% | 1.88× | 70.8% | 17.3% | 0.535 |
| 3 | 500 | 22 | 4.4% | 0.88× | 79.6% | 27.4% | 0.522 |
| 4 | 500 | 21 | 4.2% | 0.84× | 88.0% | 37.5% | 0.505 |
| 5 | 500 | 15 | 3.0% | 0.60× | 94.0% | 47.7% | 0.463 |
| 6 | 500 | 8 | 1.6% | 0.32× | 97.2% | 58.0% | 0.392 |
| 7 | 500 | 4 | 0.8% | 0.16× | 98.8% | 68.5% | 0.303 |
| 8 | 500 | 2 | 0.4% | 0.08× | 99.6% | 79.0% | 0.206 |
| 9 | 500 | 1 | 0.2% | 0.04× | 100.0% | 89.5% | 0.105 |
| 10 | 500 | 0 | 0.0% | 0.00× | 100.0% | 100.0% | 0.000 |
Sample: 5,000 accounts, 250 bad (5.0% base rate). The KS row is highlighted. Binned KS (0.535) can only ever be ≤ the exact KS (0.547), because a bin grid finds the gap at its own cut points and nowhere else — push separation down until the bars stop descending cleanly and rose bins appear, the rank-order breaks that a validation pack has to explain.
Sort every account by score, cut into equal-sized bins, and count the bads in each. The gains curve is the cumulative share of bads captured (teal) against the cumulative share of goods (indigo); the dashed vertical bar is the KS statistic, the widest gap between them. Separation moves discrimination, prevalence moves only the base rate — watch AUC and KS hold while lift collapses, because lift is a ratio to the base rate and the other two are not.
The slider that teaches the most is prevalence. Hold the model fixed and move the base rate from 5% to 25%: AUC and KS barely move, while top-decile lift collapses from 5.20× to 3.22×. Lift is a ratio to the base rate, and at prevalence it can never exceed , because a bin can only hold so many of a plentiful class. Once a quarter of the book is bad, 4× is the arithmetic ceiling no matter how good the ranking is. Anyone quoting "5× lift" without the prevalence it was measured at has quoted nothing.
Two more readings worth having reflexes for:
- Rank-order breaks: a decile riskier than the one above it. In the middle deciles that usually means a feature is non-monotone in risk. In the bottom deciles it usually means too few bads to estimate a rate at all, so check counts before writing the finding.
- Binned versus exact KS: a KS read off a ten-row table can only find the gap at the table's own cut points, so it always sits at or below the KS computed over every distinct score. Do not compare one team's binned KS to another's exact one.
Exercise
A fraud model is deployed unchanged on a riskier segment: the base rate goes from 5% to 25%, and the model's AUC is the same on both. What happens to its top-decile lift?
Need a hint?
Lift is the bin's bad rate divided by the base rate. How many bads can 10% of the population physically hold?
Is the difference real?
Model A scores 0.86, model B scores 0.84. Both numbers come from one test set, and both models scored the same examples, so their errors are correlated: an example that is hard for A is usually hard for B. That correlation is the whole game.
Compare the two AUCs as though they came from independent evaluations and you silently drop the last term, inflating the variance for nothing. You already have the paired predictions; the covariance is free evidence.
DeLong's test gives that covariance in closed form for AUCs, by writing each AUC as an average of per-subject placement values and taking their sample covariance. On a ten-subject example with and , the two AUC estimates correlate at 0.92, and the paired test returns , . Treat the same two numbers as independent and the standard error triples: , , "no evidence of a difference". Identical data, opposite decision.
McNemar's test is the fixed-threshold counterpart. Cross-tabulate the two models' correct/wrong verdicts and throw away both cells where they agree, because agreement carries no information about which is better. Only the discordant counts and remain, and under the null each discordant example is a coin flip:
With and out of 400 examples: , . The verdict rests on the 35 examples where the models disagreed, which is the correct denominator, and the reason McNemar finds differences a two-proportion test cannot.
For any other metric (precision at , F1, expected cost, NDCG) use the paired bootstrap: resample examples with replacement, score both models on the same resample, and read the percentile interval of the difference. Or the permutation test: swap the two models' predictions within each example at random and count how often the shuffled gap reaches the observed one.
Report intervals, and remember the test set is finite
"AUC difference 0.021, 95% CI [0.003, 0.039]" is a usable sentence. "p = 0.03" is not, and "significant" is less. And every comparison on the same test set spends credibility: twenty candidates at α = 0.05 produce a spurious winner on average even if all twenty are identical. Correct with Holm or Benjamini–Hochberg, and keep a final held-out set you score exactly once.
Exercise
Two classifiers were run at a fixed threshold on the same 400-example test set. A is right where B is wrong on 25 examples, B is right where A is wrong on 10, and they agree on the other 365. Which test decides whether A is genuinely more accurate?
Need a hint?
Which examples actually carry information about which model is better?
A validation pack, end to end
Put together, the four exhibits below are what a model review expects to see, and the order matters: discrimination first, then where it lands, then stability, then whether the comparison holds up.
- Discrimination: c-statistic / Gini with a confidence interval, plus the ROC and PR curves. One number, honestly bounded.
- Rank ordering: the decile table with lift, gains and KS, and an explicit note on any monotonicity break.
- Calibration: a reliability curve and ECE, plus Hosmer–Lemeshow read as a per-bin residual table rather than a verdict. A model can rank perfectly and still price everything 2× too high.
- Comparison: the paired test against the incumbent, with an effect size and an interval, on a test set scored once.
- Stability: PSI on the score and top features against the development sample, on frozen development bins.
The one-sentence summary you should be able to write
"On 14,200 held-out accounts, the new model scores c = 0.847 (95% CI 0.831 to 0.863) against the incumbent's 0.826, a paired DeLong difference of 0.021 (CI 0.009 to 0.033, p = 0.002), captures 61% of bads in the top two deciles against 55%, and is calibrated within 1.2 points of observed in every decile." Every clause of that sentence comes from one of the five exhibits above.
Common mistakes
- Quoting gamma, or a decile-table Gini, as the model's AUC. One deletes ties, the other is a ten-point trapezoid approximation.
- Comparing lift across populations with different base rates. Lift is normalised by prevalence, so the comparison is meaningless without it.
- Computing concordance on binned scores. You are then measuring your binning, not the model.
- Unpaired tests on a shared test set. Throws away the covariance term and usually the finding with it.
- Re-cutting decile boundaries every monitoring period. A population shift then shows up as a risk change. Freeze the development cut points.
- Treating "not significant" as "no difference". Usually it means the test set is too small to resolve the gap.
Key takeaways
- Concordance measures ranking without a threshold. is the AUC; Somers' D and Gini are ; gamma deletes ties and flatters coarse scores.
- The decile table says where the risk landed. Lift, gains and KS answer the operational question that a single AUC cannot, and lift depends on prevalence in a way the other two do not.
- Binning costs measurable discrimination, and the statistics disagree about how to charge for it. Compute on raw scores.
- Model comparison is a paired problem. DeLong for AUCs, McNemar at a fixed threshold, the paired bootstrap or a permutation test for anything else.
- Report effect sizes with intervals, correct for multiple comparisons, and keep one test set you score exactly once.
Related concepts
- Classification & Regression Metrics: the threshold metrics this lesson complements
- Validation Strategies: the splits these tests are run on, and why fold scores are correlated
- Calibration & Uncertainty: the audit a high c-statistic does not cover
- Concordance and rank statistics: pair counting, ties, and the rank-sum shortcut
- Decile Analysis, Gains and Lift: the full table, KS bands and Hosmer–Lemeshow
- Comparing Model Performance: DeLong, McNemar, paired resampling and multiple-comparison corrections