Is That Difference Real? DeLong, McNemar and Paired Resampling
Model A scores 0.86 AUC on the test set, model B scores 0.84. Ship A?
Not yet. Both numbers are estimates from one finite sample, and the question is whether a 0.02 gap is bigger than the noise in that sample. The answer depends almost entirely on one fact that gets thrown away constantly: the two models were evaluated on the same examples, so their errors are correlated, and a test that ignores the pairing wastes most of the evidence you have.
The one thing that matters: pairing
If you compare two AUCs as if they came from independent samples, the variance of the difference is . When the same subjects were scored twice, the correct variance is
and that covariance is typically large and positive, because a hard example is hard for both models. The covariance term is free evidence: you already have the paired predictions. Throwing it away is the single most common way a real improvement gets declared "not significant".
DeLong's test for two correlated AUCs
DeLong, DeLong and Clarke-Pearson (1988) give the covariance in closed form, using the fact that the AUC is a two-sample U-statistic. Define
For model with cases and controls, the placement values are
is the fraction of controls that case beats; is the fraction of cases that beat control . Both average to the AUC, which is what makes the decomposition work. Let and be the sample covariance matrices of these vectors across models (over the cases and the controls respectively). Then
and is asymptotically standard normal under the null of equal AUCs.
DeLong's test — is that AUC gap real, or did the test set get lucky?
Two models, 10 subjects, and one unavoidable fact: both models scored the **same** subjects, so their AUC estimates are not independent. Counting concordant pairs gives AUC_A = 19/24 = 0.7917 and AUC_B = 14/24 = 0.5833, a gap of 0.2083. The whole question is whether a gap that size could come out of a test set this small by luck.
| subject | outcome | model A | model B |
|---|---|---|---|
| s1 | case | 0.90 | 0.70 |
| s2 | case | 0.75 | 0.65 |
| s3 | case | 0.60 | 0.45 |
| s4 | case | 0.35 | 0.25 |
| s5 | control | 0.80 | 0.85 |
| s6 | control | 0.55 | 0.60 |
| s7 | control | 0.40 | 0.50 |
| s8 | control | 0.30 | 0.40 |
| s9 | control | 0.20 | 0.30 |
| s10 | control | 0.10 | 0.15 |
A two-sample test would ask whether two independent samples differ. This is a paired problem: subject s5 is a hard control for both models, and that shared difficulty is shared noise. DeLong's contribution is an exact expression for the covariance that noise creates.
Two models scored on the same ten subjects: AUC 0.7917 vs 0.5833. Watch the AUC decompose into placement values, the placement values into the S10 / S01 covariance matrices, and those into a standard error for the gap. The payoff deletes the covariance term — the one thing pairing buys — and the same 0.208 gap goes from p = 0.021 to p = 0.431.
The trace runs it on ten subjects, four cases and six controls, scored by both models: , . What to notice:
- The AUC decomposes into one placement value per subject, and those per-subject contributions are what the covariance is computed from.
- here, because both models find the same control (s5) hard.
- The paired test gives , . The payoff frame drops the covariance term and the identical 0.208 gap becomes , . Same data, opposite decision.
DeLong compares ranking only
A significant DeLong result says model A orders cases better. It says nothing about calibration, nothing about which model is better at your operating threshold, and nothing about cost. A model can win on AUC and lose on every decision you actually make with it.
McNemar's test for two classifiers at a fixed threshold
Once you have fixed a threshold, each model either gets an example right or wrong, and the comparison becomes a 2×2 table of agreement:
| B correct | B wrong | |
|---|---|---|
| A correct | ||
| A wrong |
The cells and are examples both models agree on. They carry no information about which is better, and McNemar's test discards them entirely. Only the discordant pairs matter: under the null that the two models are equally accurate, each discordant example is a coin flip, so .
Worked example. On a 400-example test set, A is right and B wrong on , A wrong and B right on , and they agree on the other 365.
The exact binomial version, , agrees closely. Without the continuity correction and , which is anticonservative on small counts. Use the exact binomial test whenever , where the approximation is unreliable.
Note what the test ignores: 365 of the 400 examples. That is the point. An accuracy difference of 3.75 percentage points is being judged on the 35 examples where the models actually disagree, which is the correct denominator for the question "is A better than B", and the reason McNemar has real power on tests where a two-proportion z-test has almost none.
Paired bootstrap and permutation: any metric at all
DeLong covers the AUC and McNemar covers accuracy at a threshold. For everything else, precision at , F1, expected cost, NDCG, a bespoke business metric, resample.
Paired bootstrap. Resample examples (not predictions) with replacement, preserving the pairing, and recompute both metrics on each resample:
- Draw example indices with replacement, stratified by class to keep the base rate stable.
- Compute on that resample, using the same indices for both models.
- Repeat 2,000 to 10,000 times. The percentile interval of is the confidence interval for the difference.
Permutation (randomisation) test. Under the null that the two models are exchangeable, swapping A's and B's prediction for a given example should change nothing. So: for each example, independently swap the two models' predictions with probability 0.5, recompute the difference, and count how often reaches the observed value. This tests exchangeability directly and needs no distributional assumption.
On the same ten-subject data as the DeLong trace, all three approaches, plus the unpaired comparison for contrast:
| Method | se of the gap | p-value | Verdict at α = 0.05 |
|---|---|---|---|
| DeLong (paired, asymptotic) | 0.090 | 0.021 | different |
| Exact permutation (all swaps) | — | 0.016 | different |
| Paired bootstrap (20,000 resamples) | 0.109 | 0.079 | can't tell |
| Treated as independent samples | 0.265 | 0.431 | can't tell |
The first two agree; the bootstrap is conservative here because with four cases the resampled difference takes only a handful of distinct values and a meaningful share of resamples land exactly at zero. That is a small-sample artefact, and the three paired methods converge as grows. The last row is the one that is simply wrong, and it is the one people report.
Report the interval, not the star
"AUC difference 0.021, 95% CI [0.003, 0.039]" tells a reader the effect could be negligible or could be worth shipping. "p = 0.03" tells them almost nothing, and "significant" tells them less. The interval also survives the reader disagreeing with your α.
Comparing across cross-validation folds
Fold-level scores are not independent: every pair of training folds overlaps in most of its data, so the usual paired t-test on fold differences has a variance estimate that is badly biased downward and a false-positive rate several times its nominal level.
Two workable options:
- Nadeau–Bengio corrected resampled t-test: inflate the variance by the train/test ratio, , then use the usual statistic. Simple, and much better calibrated than the naive version.
- 5×2cv paired t-test (Dietterich, 1998): five replications of 2-fold CV, where within each replication the two folds are disjoint. The statistic uses the first replication's difference over the mean variance of the five, with 5 degrees of freedom.
Whichever you use, a difference that only appears after fold-averaging and is smaller than the fold-to-fold spread is not a difference. See validation strategies for the splits themselves.
Many comparisons, one test set
Every test on the same test set spends some of its credibility. Evaluate 20 candidate models at α = 0.05 and you expect one spurious "winner" even if all 20 are identical.
- Bonferroni: compare each p-value against . Safe, and badly underpowered once is large.
- Holm–Bonferroni: sort the p-values, compare the smallest against , the next against , and stop at the first failure. Uniformly more powerful than Bonferroni, same guarantee.
- Benjamini–Hochberg: controls the false discovery rate instead of the family-wise error rate. The right choice when you are screening many models and can tolerate some false leads.
- A held-out final test set that is scored exactly once, after the winner is chosen on validation data. This is the only real protection against the adaptive overfitting that repeated testing causes, and no correction substitutes for it.
Common mistakes
- Two independent-sample tests instead of one paired test. The single largest source of missed improvements, as the DeLong payoff frame shows.
- Bootstrapping the two models separately. Resample the examples once per iteration and score both models on that resample; separate resamples destroy the pairing you are trying to exploit.
- Reporting a p-value with no effect size. Statistical significance at includes differences far too small to act on.
- Testing on the set you tuned on. After 200 hyperparameter evaluations, the test set's p-values are decoration.
- Reading "not significant" as "no difference". It usually means the test set is too small to resolve the gap, which is a statement about your evaluation, not about the models.
Related concepts
- Rank Ordering & Model Comparison: the lesson this page supports
- Validation Strategies: the splits these tests run on, and why fold scores are correlated
- The Bootstrap: the resampling machinery behind the paired interval, including BCa
- Concordance and rank statistics: the c-statistic whose difference DeLong is testing
- Hypothesis Testing: null distributions, p-values and what they do and do not license