Classification & Regression Metrics
This whole course is the evaluation stage of the ML project loop — the slot that decides whether anything you built in the other five actually works. And it starts with a warning: accuracy is not enough. A classifier that always predicts the majority class on a 99/1 imbalanced dataset achieves 99% accuracy while being completely useless. This lesson gives you the full toolkit for measuring model performance honestly.
The confusion matrix
Every binary classifier's performance on a labelled dataset can be summarized in a 2×2 confusion matrix:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | True Positive (TP) | False Negative (FN) |
| Actual Negative | False Positive (FP) | True Negative (TN) |
Worked example: A spam filter tested on 100 emails (70 spam, 30 legitimate):
| Predicted Spam | Predicted Not Spam | |
|---|---|---|
| Actual Spam | TP = 60 | FN = 10 |
| Actual Legit | FP = 5 | TN = 25 |
From this matrix, four metrics:
Precision and recall
Precision and recall measure different aspects of classifier performance:
-
Precision: "Of the positive predictions I made, what fraction were actually positive?" High precision = low false alarm rate. Important when false positives are costly (spam filter: legitimate email going to spam).
-
Recall (sensitivity/TPR): "Of all the actual positives, what fraction did I catch?" High recall = low miss rate. Important when false negatives are costly (cancer screening: missing a real tumor).
The precision-recall tradeoff: Most classifiers output a score, not a hard label. By raising the classification threshold (require higher confidence to predict positive), you get higher precision but lower recall. Lowering the threshold does the reverse.
Choosing the right metric
If missing positives is catastrophic (disease screening, fraud detection), optimize for recall. If false alarms are costly (email filters, automated actions), optimize for precision. F1 is the balanced default when you have no prior reason to prefer one.
F1 score and F-beta
F1 is the harmonic mean of precision and recall:
The harmonic mean penalizes extreme imbalance: if either P or R is near zero, F1 drops sharply. The arithmetic mean would be more lenient.
Worked example: TP=90, FP=10, FN=30:
F-beta generalizes with weight :
- (e.g., ): weights precision more — use when false alarms hurt
- (e.g., ): weights recall more — use when misses hurt
ROC curve and AUC
The ROC curve (Receiver Operating Characteristic) plots True Positive Rate vs False Positive Rate at every possible threshold:
Key reference points:
| Point | Interpretation |
|---|---|
| (0, 0) | Threshold = 1: predicts nothing positive |
| (1, 1) | Threshold = 0: predicts everything positive |
| (0, 1) | Perfect classifier |
| Diagonal | Random classifier (AUC = 0.5) |
AUC (Area Under the ROC Curve) summarizes the curve in one number. AUC = 0.5 → random; AUC = 1.0 → perfect. Interpretation: the probability that the classifier ranks a random positive example higher than a random negative.
ROC is misleading under severe class imbalance
With 99% negatives, FPR = FP/(FP+TN) stays small even with many false positives because TN is huge. A model can look great on ROC while producing far more false positives than true positives. In this regime, use the PR curve instead.
Precision-Recall curve
The PR curve plots Precision vs Recall at every threshold, with no involvement of TN. It is sensitive to the composition of positive predictions.
No-skill baseline: A random classifier that predicts positive with rate (the prevalence) has at all thresholds. A useful model must stay above this horizontal line.
For imbalanced datasets (rare diseases, fraud), PR-AUC is the preferred summary: it directly measures whether predicted positives are real (precision) and whether real positives are found (recall).
Everything above is one object
The confusion matrix, precision, recall, F1, the ROC curve and the PR curve are not six things to memorise. They are six readings of a single decision (where you put the threshold) taken on a single model.
One threshold, and what imbalance does to the metrics that read it
| pred + | pred − | |
| actual + | 3,967 | 1,033 |
| actual − | 1,050 | 3,950 |
Balanced classes. Slide the threshold: precision and recall move against each other, and the teal point walks along both curves at once. Now change the prevalence — the model does not change.
900 positives and 900 negatives scored by the same fixed model. The threshold slider moves the cut; the prevalence buttons change only the class mix, never the model. Each histogram is normalised to its own height — at 1% prevalence the positive class would otherwise be one invisible pixel — so the real mix lives in the confusion-matrix counts.
The model never changes: 900 positives and 900 negatives, scored once, separated well enough to reach an AUC of . Only the threshold and the class mix move.
Slide the threshold first. The teal point walks along both curves simultaneously, and precision and recall trade against each other exactly as described above — that tradeoff is not a property of your model, it is a property of the cut you chose in it.
Then press 1% and change nothing else. This is the imbalance warning above, made checkable:
| prevalence | AUC-ROC | PR-AUC | no-skill floor |
|---|---|---|---|
| 50% | 0.8727 | 0.8699 | 0.50 |
| 10% | 0.8727 | 0.5247 | 0.10 |
| 1% | 0.8727 | 0.1905 | 0.01 |
AUC-ROC is not similar across those rows. It is identical to four decimals, because TPR and FPR are both computed within a class and neither one ever sees the class mix. The PR curve collapses over the same three rows, because precision is the one metric that asks "of the things I flagged, how many were real?"
At 1% prevalence and a threshold of , this model flags 2,079 false positives to catch 79 true ones — precision . Accuracy still reads (all but unchanged from the it reported on balanced data) and the ROC curve does not move at all. Two of the three headline numbers are telling you nothing is wrong.
from sklearn.metrics import roc_auc_score, average_precision_score
roc_auc = roc_auc_score(y_true, y_scores) # AUC-ROC
pr_auc = average_precision_score(y_true, y_scores) # AUC-PR
Regression metrics
For regression tasks, the most common metrics are:
MAE vs MSE: MSE squares the residuals, giving disproportionate weight to large errors (outliers). Use MAE when outliers are common and you don't want them to dominate. Use RMSE when large errors are especially bad.
R² measures explained variance: R²=1 is perfect fit, R²=0 means the model is no better than predicting the mean, R² < 0 is possible (worse than mean).
Multi-class metrics
For classes, average precision/recall across classes:
- Macro average: compute metric per class, take unweighted mean — treats all classes equally regardless of size
- Micro average: pool all TP/FP/FN across classes, then compute — dominated by frequent classes
- Weighted average: weight each class by its support — best when class imbalance is present
For the geometric derivation of the ROC curve and the proof that AUC equals the rank-sum probability, see the deep dive:
Exercise
A classifier has 90 true positives, 10 false positives, and 30 false negatives. What is its F1 score?
Need a hint?
Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2·P·R/(P+R).
Exercise
For a rare-disease screening test (1% prevalence), why is accuracy a misleading metric and what should you use instead?
Need a hint?
What accuracy does a classifier that always predicts 'healthy' achieve?
Exercise
You have a highly imbalanced binary dataset (0.5% positive rate). Should you use ROC-AUC or PR-AUC to compare two classifiers, and why?
Need a hint?
ROC uses true negative rate; PR does not — which one can a classifier 'game' by exploiting the large negative class?
Common mistakes
- Reporting only accuracy on imbalanced data: a 99% accuracy can mean the classifier does nothing useful. Always check precision/recall/F1 too.
- Choosing the threshold after seeing test data: threshold should be tuned on the validation set, not the test set.
- Conflating AUC-ROC and AUC-PR: for imbalanced data they can diverge dramatically. A model with AUC-ROC=0.95 can have AUC-PR=0.30.
- Using R² alone for regression: a high R² can coexist with large systematic errors. Always plot residuals.
- Macro averaging with very imbalanced classes: if one class has 5 samples, a bad macro-average can be dominated by noise on that class.
Key takeaways
- The confusion matrix (TP/FP/TN/FN) is the foundation of all classification metrics.
- Precision = "what fraction of my positives are real"; Recall = "what fraction of real positives did I find." They trade off at each threshold.
- F1 is the harmonic mean of P and R; F-beta lets you weight one more heavily.
- ROC-AUC measures ranking quality across all thresholds; AUC = P(positive ranked above negative).
- PR-AUC is preferred for severe class imbalance — it ignores the huge true-negative pool that inflates ROC.
- RMSE penalizes outliers more than MAE; R² measures explained variance relative to the mean predictor.
Related concepts
- Logistic Regression: the model whose output feeds into these metrics
- Validation Strategies: how to measure these metrics without leaking test data
- Rank Ordering & Model Comparison: the same classifier measured without a threshold, and tested against a rival
- Concordance and rank statistics: AUC as a pair count, plus Somers' D, Gini and the tie conventions
- The ML Project Loop: where evaluation sits in the six-stage project spine, and why the metric you pick is really the objective in disguise