ROC Curve and AUC: Derivation and Rank-Sum InterpretationOpen in Colab

ROC Curve and AUC

ROC-AUC is the most common threshold-independent metric for binary classifiers. This page derives the curve construction from scratch and establishes the probabilistic rank-sum interpretation that makes AUC meaningful beyond a raw number.

Constructing the ROC curve step by step

A classifier produces a score s(x)∈Rs(x) \in \mathbb{R} for each example. Sweeping a threshold τ\tau from +∞+\infty down to −∞-\infty traces a path through (FPR, TPR) space. The ROC curve is the full path.

TPR(τ)=P(y^=1∣y=1)=TP(τ)n+,FPR(τ)=P(y^=1∣y=0)=FP(τ)n−\text{TPR}(\tau) = P(\hat{y}=1 \mid y=1) = \frac{\text{TP}(\tau)}{n_+}, \qquad \text{FPR}(\tau) = P(\hat{y}=1 \mid y=0) = \frac{\text{FP}(\tau)}{n_-}

Algorithm (exact, O(nlog⁡n)O(n\log n)):

  1. Sort all nn examples by descending score.
  2. Initialize (FPR,TPR)=(0,0)(\text{FPR}, \text{TPR}) = (0, 0).
  3. Process examples in sorted order: a positive increments TPR by 1/n+1/n_+; a negative increments FPR by 1/n−1/n_-.
  4. Record each (FPR, TPR) point and connect the dots.

Worked example (n+=3n_+ = 3, n−=2n_- = 2):

Sorted scores and labels: (0.92,+),  (0.85,+),  (0.61,−),  (0.43,+),  (0.18,−)(0.92, +),\; (0.85, +),\; (0.61, -),\; (0.43, +),\; (0.18, -)

StepScoreLabelFPRTPR
Start——00
10.92+01/3
20.85+02/3
30.61−1/22/3
40.43+1/21
50.18−11

AUC via the trapezoid rule ∑(FPRi+1−FPRi)⋅TPRi+TPRi+12\sum (FPR_{i+1} - FPR_i) \cdot \frac{TPR_i + TPR_{i+1}}{2}:

  • Segment (0, 2/3) → (1/2, 2/3): ΔFPR=1/2\Delta\text{FPR} = 1/2, avg TPR =2/3= 2/3. Contribution: 1/31/3.
  • Segment (1/2, 1) → (1, 1): ΔFPR=1/2\Delta\text{FPR} = 1/2, avg TPR =1= 1. Contribution: 1/21/2.

AUC=13+12=56≈0.833\text{AUC} = \frac{1}{3} + \frac{1}{2} = \frac{5}{6} \approx 0.833

Watch it run

The same five examples, one threshold step at a time — then the two things the sections below formalize, shown rather than argued:

ROC — sweeping a threshold, and what class imbalance does to it

1/9

Five scored examples — three positives and two negatives — sorted by descending score. ROC never looks at the scores themselves, only at the *order* they induce: the whole curve is traced by sweeping a threshold from +∞ down to −∞ and recording where each example falls.

python
1def roc(scores, labels):
2 order = argsort(scores)[::-1] # high first
3 n_pos = sum(labels)
4 n_neg = len(labels) - n_pos
5 tp = fp = 0
6 pts = [(0.0, 0.0)]
7 for i in order:
8 if labels[i] == 1:
9 tp += 1 # climb: TPR only
10 else:
11 fp += 1 # step right: FPR only
12 pts.append((fp / n_neg, tp / n_pos))
13 auc = 0.0
14 for (x0, y0), (x1, y1) in zip(pts, pts[1:]):
15 # trapezoid under this segment
16 auc += (x1 - x0) * (y0 + y1) / 2
17 return pts, auc
state
sorted by score
0.92+
0.85+
0.61−
0.43+
0.18−
the curve, empty (dashed = random classifier)
0110FPRTPR

Because only the ranking matters, any monotone rescaling of the scores — a sigmoid, a temperature change, a calibration map — leaves the ROC curve and the AUC completely unchanged.

The ROC curve built one example at a time on the five-point worked example above: a positive moves the curve up, a negative moves it right, and AUC is the trapezoid area — 5/6, cross-checked against the Wilcoxon concordant-pair count on the same data. The final step replicates every negative 100× without changing the ranking at all: AUC does not move a digit, while precision at full recall collapses by more than 25×.

What to notice as you step:

  • The scores never enter the arithmetic. Only their order does, which is why any monotone rescaling — a sigmoid, a temperature change, a calibration map — leaves the curve and the AUC bit-identical. AUC measures ranking, and is blind to calibration by construction.
  • Positives move the curve up, negatives move it right. A perfect ranker goes straight up and then straight across; the diagonal is a coin flip.
  • Step 8 reaches the same AUC by counting pairs: 5 of the 6 positive–negative pairs are correctly ordered, so AUC = 5/6. That is the Wilcoxon–Mann–Whitney equivalence the next section proves.
  • The last step replicates every negative 100× without touching the ranking. AUC does not move a single digit, because FPR divides by n−n_- and the scaling cancels. Precision divides by TP + FP, which does not cancel, so precision at full recall drops from 0.75 to 0.029. Same model, same ordering, 26× worse where it matters operationally.

The rank-sum interpretation

Theorem (Wilcoxon-Mann-Whitney equivalence):

AUC=P ⁣(s(x+)>s(x−))\text{AUC} = P\!\left(s(x^+) > s(x^-)\right)

where x+x^+ is drawn uniformly from the positives and x−x^- from the negatives.

Proof sketch: AUC =∫01TPR(t) d FPR(t)= \int_0^1 \text{TPR}(t)\,d\,\text{FPR}(t). By change of variables, this integral counts the fraction of (x+,x−)(x^+, x^-) pairs where the positive scores higher — exactly the Wilcoxon-Mann-Whitney U statistic normalized by n+n−n_+ n_-.

Verification from the worked example: Total pairs =3×2=6= 3 \times 2 = 6. Concordant pairs (positive scores above negative):

PositiveNegativeConcordant?
0.920.61✓
0.920.18✓
0.850.61✓
0.850.18✓
0.430.61✗
0.430.18✓

Concordant = 5. AUC =5/6≈0.833= 5/6 \approx 0.833. ✓ Matches the curve construction.

What this means in practice: AUC = 0.85 means the classifier ranks a random positive above a random negative 85% of the time. AUC measures ranking quality, not calibration — a model can have perfect AUC while being systematically overconfident.

PR curve and PR-AUC

The Precision-Recall curve plots Precision vs Recall as τ\tau varies, with no role for true negatives:

Precision(τ)=TPTP+FP,Recall(τ)=TPn+\text{Precision}(\tau) = \frac{\text{TP}}{\text{TP} + \text{FP}}, \qquad \text{Recall}(\tau) = \frac{\text{TP}}{n_+}

Why it matters for class imbalance: with n−≫n+n_- \gg n_+, FPR =FP/n−= \text{FP} / n_- stays tiny even with many false positives (because n−n_- is huge). A model can achieve ROC-AUC ≈0.95\approx 0.95 while PR-AUC is near the no-skill baseline of n+/nn_+ / n.

No-skill baseline: a random classifier predicts positive at rate p=n+/np = n_+ / n for all thresholds, yielding constant Precision=p\text{Precision} = p. Any useful model must beat this horizontal line.

ROC-AUC is misleading under severe imbalance

With 1% positives, even a model that generates 100× more false positives than true positives can show ROC-AUC > 0.9 because FPR is computed against the enormous TN pool. PR-AUC exposes this directly — it would reflect the low precision. Use ROC-AUC for balanced data; use PR-AUC when the positive class is rare (fraud, disease, defect detection).

Python
from sklearn.metrics import roc_auc_score, average_precision_score

roc_auc = roc_auc_score(y_true, y_scores)          # AUC-ROC
pr_auc  = average_precision_score(y_true, y_scores) # AUC-PR (also called AP)

Threshold selection

AUC summarizes the entire curve, but deployment requires one threshold. Common criteria:

CriterionDefinitionUse case
Youden's Jarg⁡max⁡τ(TPR−FPR)\arg\max_\tau (\text{TPR} - \text{FPR})Maximize overall separation
Cost-sensitivearg⁡min⁡τ(cFP⋅FP+cFN⋅FN)\arg\min_\tau (c_\text{FP} \cdot \text{FP} + c_\text{FN} \cdot \text{FN})Asymmetric error costs (fraud, medical)
F1-optimalarg⁡max⁡τF1(τ)\arg\max_\tau F_1(\tau)Balanced precision / recall
Fixed recallmin⁡τ\min \tau s.t. Recall≥r\text{Recall} \ge rGuaranteed sensitivity (safety screening)
Fixed FPRmax⁡τ\max \tau s.t. FPR≤f\text{FPR} \le fBounded false alarm rate

Always tune the threshold on the validation set, never on the test set — the test set gives you a point estimate of expected performance, not a tuning knob.

Common pitfalls

  1. Reporting only ROC-AUC on imbalanced data: a 0.95 AUC can coexist with near-baseline PR-AUC. Always report both.
  2. Treating AUC as calibration: AUC measures rank ordering. A perfectly calibrated model and a wildly overconfident model can have the same AUC if they rank examples identically.
  3. Optimizing threshold on the test set: leaks test data; always use the validation set.
  4. Multi-class AUC without specifying the averaging scheme: sklearn's roc_auc_score defaults to macro one-vs-rest; document which you used.
  5. Wide cross-validation variance: a single train/test split gives a noisy AUC estimate. Report mean ± std across kk folds.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
ROC Curve and AUC: Derivation and…Concordance: the c-statis…Platt Scaling and Isotoni…Classification & Regressi…Validation StrategiesCalibration & Uncertainty