Is That Difference Real? DeLong, McNemar and Paired ResamplingOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Is That Difference Real? DeLong, McNemar and Paired Resampling

Model A scores 0.86 AUC on the test set, model B scores 0.84. Ship A?

Not yet. Both numbers are estimates from one finite sample, and the question is whether a 0.02 gap is bigger than the noise in that sample. The answer depends almost entirely on one fact that gets thrown away constantly: the two models were evaluated on the same examples, so their errors are correlated, and a test that ignores the pairing wastes most of the evidence you have.

The one thing that matters: pairing

If you compare two AUCs as if they came from independent samples, the variance of the difference is var⁡(A^)+var⁡(B^)\operatorname{var}(\hat{A}) + \operatorname{var}(\hat{B}). When the same subjects were scored twice, the correct variance is

var⁡(A^−B^)=var⁡(A^)+var⁡(B^)−2cov⁡(A^,B^)\operatorname{var}(\hat{A} - \hat{B}) = \operatorname{var}(\hat{A}) + \operatorname{var}(\hat{B}) - 2\operatorname{cov}(\hat{A}, \hat{B})

and that covariance is typically large and positive, because a hard example is hard for both models. The covariance term is free evidence: you already have the paired predictions. Throwing it away is the single most common way a real improvement gets declared "not significant".

DeLong's test for two correlated AUCs

DeLong, DeLong and Clarke-Pearson (1988) give the covariance in closed form, using the fact that the AUC is a two-sample U-statistic. Define

ψ(x,y)={1x>y12x=y0x<y\psi(x, y) = \begin{cases} 1 & x > y \\ \tfrac{1}{2} & x = y \\ 0 & x < y \end{cases}

For model rr with mm cases and nn controls, the placement values are

V10r(pi)=1n∑j=1nψ(pi,qj),V01r(qj)=1m∑i=1mψ(pi,qj)V_{10}^{r}(p_i) = \frac{1}{n}\sum_{j=1}^{n} \psi(p_i, q_j), \qquad V_{01}^{r}(q_j) = \frac{1}{m}\sum_{i=1}^{m} \psi(p_i, q_j)

V10(pi)V_{10}(p_i) is the fraction of controls that case ii beats; V01(qj)V_{01}(q_j) is the fraction of cases that beat control jj. Both average to the AUC, which is what makes the decomposition work. Let S10S_{10} and S01S_{01} be the sample covariance matrices of these vectors across models (over the mm cases and the nn controls respectively). Then

S=S10m+S01n,z=A^−B^SAA+SBB−2SABS = \frac{S_{10}}{m} + \frac{S_{01}}{n}, \qquad z = \frac{\hat{A} - \hat{B}}{\sqrt{S_{AA} + S_{BB} - 2S_{AB}}}

and zz is asymptotically standard normal under the null of equal AUCs.

DeLong's test — is that AUC gap real, or did the test set get lucky?

1/7

Two models, 10 subjects, and one unavoidable fact: both models scored the **same** subjects, so their AUC estimates are not independent. Counting concordant pairs gives AUC_A = 19/24 = 0.7917 and AUC_B = 14/24 = 0.5833, a gap of 0.2083. The whole question is whether a gap that size could come out of a test set this small by luck.

python
1def psi(x, y):
2 # win 1, tie 1/2, loss 0
3 if x > y: return 1.0
4 return 0.5 if x == y else 0.0
5
6def placements(pos, neg):
7 # V10[i]: case i vs every control
8 v10 = [avg([psi(p, q) for q in neg])
9 for p in pos]
10 # V01[j]: control j vs every case
11 v01 = [avg([psi(p, q) for p in pos])
12 for q in neg]
13 return v10, v01
14
15def delong(models, m, n):
16 A, V10, V01 = [], [], []
17 for pos, neg in models:
18 a, b = placements(pos, neg)
19 V10.append(a); V01.append(b)
20 A.append(avg(a)) # = AUC
21 S10 = cov(V10, ddof=1) # m cases
22 S01 = cov(V01, ddof=1) # n controls
23 S = S10 / m + S01 / n
24 var = S[0][0] + S[1][1] - 2 * S[0][1]
25 z = (A[0] - A[1]) / sqrt(var)
26 return z, 2 * (1 - Phi(abs(z)))
state
the same ten subjects, scored twice
subjectoutcomemodel Amodel B
s1case0.900.70
s2case0.750.65
s3case0.600.45
s4case0.350.25
s5control0.800.85
s6control0.550.60
s7control0.400.50
s8control0.300.40
s9control0.200.30
s10control0.100.15
the two AUCs
cases (m) → 4controls (n) → 6AUC_A → 19/24 = 0.7917AUC_B → 14/24 = 0.5833difference → 0.2083

A two-sample test would ask whether two independent samples differ. This is a paired problem: subject s5 is a hard control for both models, and that shared difficulty is shared noise. DeLong's contribution is an exact expression for the covariance that noise creates.

Two models scored on the same ten subjects: AUC 0.7917 vs 0.5833. Watch the AUC decompose into placement values, the placement values into the S10 / S01 covariance matrices, and those into a standard error for the gap. The payoff deletes the covariance term — the one thing pairing buys — and the same 0.208 gap goes from p = 0.021 to p = 0.431.

The trace runs it on ten subjects, four cases and six controls, scored by both models: A^=19/24=0.7917\hat{A} = 19/24 = 0.7917, B^=14/24=0.5833\hat{B} = 14/24 = 0.5833. What to notice:

  • The AUC decomposes into one placement value per subject, and those per-subject contributions are what the covariance is computed from.
  • corr⁡(A^,B^)=0.921\operatorname{corr}(\hat{A}, \hat{B}) = 0.921 here, because both models find the same control (s5) hard.
  • The paired test gives z=2.31z = 2.31, p=0.021p = 0.021. The payoff frame drops the covariance term and the identical 0.208 gap becomes z=0.79z = 0.79, p=0.431p = 0.431. Same data, opposite decision.

DeLong compares ranking only

A significant DeLong result says model A orders cases better. It says nothing about calibration, nothing about which model is better at your operating threshold, and nothing about cost. A model can win on AUC and lose on every decision you actually make with it.

McNemar's test for two classifiers at a fixed threshold

Once you have fixed a threshold, each model either gets an example right or wrong, and the comparison becomes a 2×2 table of agreement:

B correctB wrong
A correctaabb
A wrongccdd

The cells aa and dd are examples both models agree on. They carry no information about which is better, and McNemar's test discards them entirely. Only the discordant pairs matter: under the null that the two models are equally accurate, each discordant example is a coin flip, so b∼Binomial(b+c,0.5)b \sim \text{Binomial}(b + c, 0.5).

χ2=(∣b−c∣−1)2b+c(with continuity correction, 1 df)\chi^2 = \frac{(|b - c| - 1)^2}{b + c} \quad \text{(with continuity correction, 1 df)}

Worked example. On a 400-example test set, A is right and B wrong on b=25b = 25, A wrong and B right on c=10c = 10, and they agree on the other 365.

χ2=(∣25−10∣−1)235=19635=5.60  ⇒  p=0.018\chi^2 = \frac{(|25 - 10| - 1)^2}{35} = \frac{196}{35} = 5.60 \;\Rightarrow\; p = 0.018

The exact binomial version, p=2⋅P(X≤10∣n=35,p=0.5)=0.0167p = 2 \cdot P(X \le 10 \mid n = 35, p = 0.5) = 0.0167, agrees closely. Without the continuity correction χ2=6.43\chi^2 = 6.43 and p=0.011p = 0.011, which is anticonservative on small counts. Use the exact binomial test whenever b+c<25b + c < 25, where the χ2\chi^2 approximation is unreliable.

Note what the test ignores: 365 of the 400 examples. That is the point. An accuracy difference of 3.75 percentage points is being judged on the 35 examples where the models actually disagree, which is the correct denominator for the question "is A better than B", and the reason McNemar has real power on tests where a two-proportion z-test has almost none.

Paired bootstrap and permutation: any metric at all

DeLong covers the AUC and McNemar covers accuracy at a threshold. For everything else, precision at kk, F1, expected cost, NDCG, a bespoke business metric, resample.

Paired bootstrap. Resample examples (not predictions) with replacement, preserving the pairing, and recompute both metrics on each resample:

  1. Draw nn example indices with replacement, stratified by class to keep the base rate stable.
  2. Compute δ(b)=MA−MB\delta^{(b)} = M_A - M_B on that resample, using the same indices for both models.
  3. Repeat 2,000 to 10,000 times. The percentile interval of {δ(b)}\{\delta^{(b)}\} is the confidence interval for the difference.

Permutation (randomisation) test. Under the null that the two models are exchangeable, swapping A's and B's prediction for a given example should change nothing. So: for each example, independently swap the two models' predictions with probability 0.5, recompute the difference, and count how often ∣δ∣|\delta| reaches the observed value. This tests exchangeability directly and needs no distributional assumption.

On the same ten-subject data as the DeLong trace, all three approaches, plus the unpaired comparison for contrast:

Methodse of the gapp-valueVerdict at α = 0.05
DeLong (paired, asymptotic)0.0900.021different
Exact permutation (all 2102^{10} swaps)—0.016different
Paired bootstrap (20,000 resamples)0.1090.079can't tell
Treated as independent samples0.2650.431can't tell

The first two agree; the bootstrap is conservative here because with four cases the resampled difference takes only a handful of distinct values and a meaningful share of resamples land exactly at zero. That is a small-sample artefact, and the three paired methods converge as nn grows. The last row is the one that is simply wrong, and it is the one people report.

Report the interval, not the star

"AUC difference 0.021, 95% CI [0.003, 0.039]" tells a reader the effect could be negligible or could be worth shipping. "p = 0.03" tells them almost nothing, and "significant" tells them less. The interval also survives the reader disagreeing with your α.

Comparing across cross-validation folds

Fold-level scores are not independent: every pair of training folds overlaps in most of its data, so the usual paired t-test on kk fold differences has a variance estimate that is badly biased downward and a false-positive rate several times its nominal level.

Two workable options:

  • Nadeau–Bengio corrected resampled t-test: inflate the variance by the train/test ratio, var⁡corr=(1k+ntestntrain)σ^2\operatorname{var}_{\text{corr}} = \left(\frac{1}{k} + \frac{n_{\text{test}}}{n_{\text{train}}}\right) \hat{\sigma}^2, then use the usual tt statistic. Simple, and much better calibrated than the naive version.
  • 5×2cv paired t-test (Dietterich, 1998): five replications of 2-fold CV, where within each replication the two folds are disjoint. The statistic uses the first replication's difference over the mean variance of the five, with 5 degrees of freedom.

Whichever you use, a difference that only appears after fold-averaging and is smaller than the fold-to-fold spread is not a difference. See validation strategies for the splits themselves.

Many comparisons, one test set

Every test on the same test set spends some of its credibility. Evaluate 20 candidate models at α = 0.05 and you expect one spurious "winner" even if all 20 are identical.

  • Bonferroni: compare each p-value against α/m\alpha / m. Safe, and badly underpowered once mm is large.
  • Holm–Bonferroni: sort the p-values, compare the smallest against α/m\alpha/m, the next against α/(m−1)\alpha/(m-1), and stop at the first failure. Uniformly more powerful than Bonferroni, same guarantee.
  • Benjamini–Hochberg: controls the false discovery rate instead of the family-wise error rate. The right choice when you are screening many models and can tolerate some false leads.
  • A held-out final test set that is scored exactly once, after the winner is chosen on validation data. This is the only real protection against the adaptive overfitting that repeated testing causes, and no correction substitutes for it.

Common mistakes

  • Two independent-sample tests instead of one paired test. The single largest source of missed improvements, as the DeLong payoff frame shows.
  • Bootstrapping the two models separately. Resample the examples once per iteration and score both models on that resample; separate resamples destroy the pairing you are trying to exploit.
  • Reporting a p-value with no effect size. Statistical significance at n=106n = 10^6 includes differences far too small to act on.
  • Testing on the set you tuned on. After 200 hyperparameter evaluations, the test set's p-values are decoration.
  • Reading "not significant" as "no difference". It usually means the test set is too small to resolve the gap, which is a statement about your evaluation, not about the models.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Is That Difference Real? DeLong, …Concordance: the c-statis…The Bootstrap: Resampling…Hypothesis Testing: Null,…Validation StrategiesRank Ordering & Model Com…