ROC Curve and AUC
ROC-AUC is the most common threshold-independent metric for binary classifiers. This page derives the curve construction from scratch and establishes the probabilistic rank-sum interpretation that makes AUC meaningful beyond a raw number.
Constructing the ROC curve step by step
A classifier produces a score for each example. Sweeping a threshold from down to traces a path through (FPR, TPR) space. The ROC curve is the full path.
Algorithm (exact, ):
- Sort all examples by descending score.
- Initialize .
- Process examples in sorted order: a positive increments TPR by ; a negative increments FPR by .
- Record each (FPR, TPR) point and connect the dots.
Worked example (, ):
Sorted scores and labels:
| Step | Score | Label | FPR | TPR |
|---|---|---|---|---|
| Start | — | — | 0 | 0 |
| 1 | 0.92 | + | 0 | 1/3 |
| 2 | 0.85 | + | 0 | 2/3 |
| 3 | 0.61 | − | 1/2 | 2/3 |
| 4 | 0.43 | + | 1/2 | 1 |
| 5 | 0.18 | − | 1 | 1 |
AUC via the trapezoid rule :
- Segment (0, 2/3) → (1/2, 2/3): , avg TPR . Contribution: .
- Segment (1/2, 1) → (1, 1): , avg TPR . Contribution: .
Watch it run
The same five examples, one threshold step at a time — then the two things the sections below formalize, shown rather than argued:
ROC — sweeping a threshold, and what class imbalance does to it
Five scored examples — three positives and two negatives — sorted by descending score. ROC never looks at the scores themselves, only at the *order* they induce: the whole curve is traced by sweeping a threshold from +∞ down to −∞ and recording where each example falls.
Because only the ranking matters, any monotone rescaling of the scores — a sigmoid, a temperature change, a calibration map — leaves the ROC curve and the AUC completely unchanged.
The ROC curve built one example at a time on the five-point worked example above: a positive moves the curve up, a negative moves it right, and AUC is the trapezoid area — 5/6, cross-checked against the Wilcoxon concordant-pair count on the same data. The final step replicates every negative 100× without changing the ranking at all: AUC does not move a digit, while precision at full recall collapses by more than 25×.
What to notice as you step:
- The scores never enter the arithmetic. Only their order does, which is why any monotone rescaling — a sigmoid, a temperature change, a calibration map — leaves the curve and the AUC bit-identical. AUC measures ranking, and is blind to calibration by construction.
- Positives move the curve up, negatives move it right. A perfect ranker goes straight up and then straight across; the diagonal is a coin flip.
- Step 8 reaches the same AUC by counting pairs: 5 of the 6 positive–negative pairs are correctly ordered, so AUC = 5/6. That is the Wilcoxon–Mann–Whitney equivalence the next section proves.
- The last step replicates every negative 100× without touching the ranking. AUC does not move a single digit, because FPR divides by and the scaling cancels. Precision divides by TP + FP, which does not cancel, so precision at full recall drops from 0.75 to 0.029. Same model, same ordering, 26× worse where it matters operationally.
The rank-sum interpretation
Theorem (Wilcoxon-Mann-Whitney equivalence):
where is drawn uniformly from the positives and from the negatives.
Proof sketch: AUC . By change of variables, this integral counts the fraction of pairs where the positive scores higher — exactly the Wilcoxon-Mann-Whitney U statistic normalized by .
Verification from the worked example: Total pairs . Concordant pairs (positive scores above negative):
| Positive | Negative | Concordant? |
|---|---|---|
| 0.92 | 0.61 | ✓ |
| 0.92 | 0.18 | ✓ |
| 0.85 | 0.61 | ✓ |
| 0.85 | 0.18 | ✓ |
| 0.43 | 0.61 | ✗ |
| 0.43 | 0.18 | ✓ |
Concordant = 5. AUC . ✓ Matches the curve construction.
What this means in practice: AUC = 0.85 means the classifier ranks a random positive above a random negative 85% of the time. AUC measures ranking quality, not calibration — a model can have perfect AUC while being systematically overconfident.
PR curve and PR-AUC
The Precision-Recall curve plots Precision vs Recall as varies, with no role for true negatives:
Why it matters for class imbalance: with , FPR stays tiny even with many false positives (because is huge). A model can achieve ROC-AUC while PR-AUC is near the no-skill baseline of .
No-skill baseline: a random classifier predicts positive at rate for all thresholds, yielding constant . Any useful model must beat this horizontal line.
ROC-AUC is misleading under severe imbalance
With 1% positives, even a model that generates 100× more false positives than true positives can show ROC-AUC > 0.9 because FPR is computed against the enormous TN pool. PR-AUC exposes this directly — it would reflect the low precision. Use ROC-AUC for balanced data; use PR-AUC when the positive class is rare (fraud, disease, defect detection).
from sklearn.metrics import roc_auc_score, average_precision_score
roc_auc = roc_auc_score(y_true, y_scores) # AUC-ROC
pr_auc = average_precision_score(y_true, y_scores) # AUC-PR (also called AP)
Threshold selection
AUC summarizes the entire curve, but deployment requires one threshold. Common criteria:
| Criterion | Definition | Use case |
|---|---|---|
| Youden's J | Maximize overall separation | |
| Cost-sensitive | Asymmetric error costs (fraud, medical) | |
| F1-optimal | Balanced precision / recall | |
| Fixed recall | s.t. | Guaranteed sensitivity (safety screening) |
| Fixed FPR | s.t. | Bounded false alarm rate |
Always tune the threshold on the validation set, never on the test set — the test set gives you a point estimate of expected performance, not a tuning knob.
Common pitfalls
- Reporting only ROC-AUC on imbalanced data: a 0.95 AUC can coexist with near-baseline PR-AUC. Always report both.
- Treating AUC as calibration: AUC measures rank ordering. A perfectly calibrated model and a wildly overconfident model can have the same AUC if they rank examples identically.
- Optimizing threshold on the test set: leaks test data; always use the validation set.
- Multi-class AUC without specifying the averaging scheme:
sklearn'sroc_auc_scoredefaults to macro one-vs-rest; document which you used. - Wide cross-validation variance: a single train/test split gives a noisy AUC estimate. Report mean ± std across folds.
Related concepts
- Classification Metrics: precision, recall, F1, confusion matrix
- Validation Strategies: how to measure metrics without leaking test data
- Calibration and Uncertainty: why AUC ≠ calibration; ECE and reliability diagrams