Classification & Regression MetricsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Classification & Regression Metrics

This whole course is the evaluation stage of the ML project loop — the slot that decides whether anything you built in the other five actually works. And it starts with a warning: accuracy is not enough. A classifier that always predicts the majority class on a 99/1 imbalanced dataset achieves 99% accuracy while being completely useless. This lesson gives you the full toolkit for measuring model performance honestly.

Deep dive
The ML Project Loop
Includes runnable notebook

The confusion matrix

Every binary classifier's performance on a labelled dataset can be summarized in a 2×2 confusion matrix:

Predicted PositivePredicted Negative
Actual PositiveTrue Positive (TP)False Negative (FN)
Actual NegativeFalse Positive (FP)True Negative (TN)

Worked example: A spam filter tested on 100 emails (70 spam, 30 legitimate):

Predicted SpamPredicted Not Spam
Actual SpamTP = 60FN = 10
Actual LegitFP = 5TN = 25

From this matrix, four metrics:

Accuracy=TP+TNN=60+25100=0.85\text{Accuracy} = \frac{TP + TN}{N} = \frac{60 + 25}{100} = 0.85 Precision=TPTP+FP=6065≈0.923\text{Precision} = \frac{TP}{TP + FP} = \frac{60}{65} \approx 0.923 Recall=TPTP+FN=6070≈0.857\text{Recall} = \frac{TP}{TP + FN} = \frac{60}{70} \approx 0.857 Specificity=TNTN+FP=2530≈0.833\text{Specificity} = \frac{TN}{TN + FP} = \frac{25}{30} \approx 0.833

Precision and recall

Precision and recall measure different aspects of classifier performance:

  • Precision: "Of the positive predictions I made, what fraction were actually positive?" High precision = low false alarm rate. Important when false positives are costly (spam filter: legitimate email going to spam).

  • Recall (sensitivity/TPR): "Of all the actual positives, what fraction did I catch?" High recall = low miss rate. Important when false negatives are costly (cancer screening: missing a real tumor).

The precision-recall tradeoff: Most classifiers output a score, not a hard label. By raising the classification threshold (require higher confidence to predict positive), you get higher precision but lower recall. Lowering the threshold does the reverse.

Choosing the right metric

If missing positives is catastrophic (disease screening, fraud detection), optimize for recall. If false alarms are costly (email filters, automated actions), optimize for precision. F1 is the balanced default when you have no prior reason to prefer one.

F1 score and F-beta

F1 is the harmonic mean of precision and recall:

F1=2⋅P⋅RP+R=2⋅TP2⋅TP+FP+FNF_1 = \frac{2 \cdot P \cdot R}{P + R} = \frac{2 \cdot TP}{2 \cdot TP + FP + FN}

The harmonic mean penalizes extreme imbalance: if either P or R is near zero, F1 drops sharply. The arithmetic mean would be more lenient.

Worked example: TP=90, FP=10, FN=30:

  • P=90/100=0.9P = 90/100 = 0.9
  • R=90/120=0.75R = 90/120 = 0.75
  • F1=2×0.9×0.75/(0.9+0.75)=1.35/1.65≈0.818F_1 = 2 \times 0.9 \times 0.75 / (0.9 + 0.75) = 1.35 / 1.65 \approx 0.818

F-beta generalizes with weight β\beta:

Fβ=(1+β2)⋅P⋅Rβ2⋅P+RF_\beta = (1 + \beta^2) \cdot \frac{P \cdot R}{\beta^2 \cdot P + R}
  • β<1\beta < 1 (e.g., F0.5F_{0.5}): weights precision more — use when false alarms hurt
  • β>1\beta > 1 (e.g., F2F_2): weights recall more — use when misses hurt

ROC curve and AUC

The ROC curve (Receiver Operating Characteristic) plots True Positive Rate vs False Positive Rate at every possible threshold:

TPR=TPTP+FN,FPR=FPFP+TN\text{TPR} = \frac{TP}{TP + FN}, \qquad \text{FPR} = \frac{FP}{FP + TN}

Key reference points:

PointInterpretation
(0, 0)Threshold = 1: predicts nothing positive
(1, 1)Threshold = 0: predicts everything positive
(0, 1)Perfect classifier
DiagonalRandom classifier (AUC = 0.5)

AUC (Area Under the ROC Curve) summarizes the curve in one number. AUC = 0.5 → random; AUC = 1.0 → perfect. Interpretation: the probability that the classifier ranks a random positive example higher than a random negative.

ROC is misleading under severe class imbalance

With 99% negatives, FPR = FP/(FP+TN) stays small even with many false positives because TN is huge. A model can look great on ROC while producing far more false positives than true positives. In this regime, use the PR curve instead.

Precision-Recall curve

The PR curve plots Precision vs Recall at every threshold, with no involvement of TN. It is sensitive to the composition of positive predictions.

No-skill baseline: A random classifier that predicts positive with rate pp (the prevalence) has Precision=p\text{Precision} = p at all thresholds. A useful model must stay above this horizontal line.

For imbalanced datasets (rare diseases, fraud), PR-AUC is the preferred summary: it directly measures whether predicted positives are real (precision) and whether real positives are found (recall).

Everything above is one object

The confusion matrix, precision, recall, F1, the ROC curve and the PR curve are not six things to memorise. They are six readings of a single decision (where you put the threshold) taken on a single model.

One threshold, and what imbalance does to the metrics that read it

0.50score 01positivesnegatives
FPRTPRROCrecallprecisionPR
pred +pred −
actual +3,9671,033
actual −1,0503,950
prevalence
precision0.791
recall0.793
F10.792
accuracy0.792
AUC-ROC0.8727
PR-AUC0.8699

Balanced classes. Slide the threshold: precision and recall move against each other, and the teal point walks along both curves at once. Now change the prevalence — the model does not change.

900 positives and 900 negatives scored by the same fixed model. The threshold slider moves the cut; the prevalence buttons change only the class mix, never the model. Each histogram is normalised to its own height — at 1% prevalence the positive class would otherwise be one invisible pixel — so the real mix lives in the confusion-matrix counts.

The model never changes: 900 positives and 900 negatives, scored once, separated well enough to reach an AUC of 0.8730.873. Only the threshold and the class mix move.

Slide the threshold first. The teal point walks along both curves simultaneously, and precision and recall trade against each other exactly as described above — that tradeoff is not a property of your model, it is a property of the cut you chose in it.

Then press 1% and change nothing else. This is the imbalance warning above, made checkable:

prevalenceAUC-ROCPR-AUCno-skill floor
50%0.87270.86990.50
10%0.87270.52470.10
1%0.87270.19050.01

AUC-ROC is not similar across those rows. It is identical to four decimals, because TPR and FPR are both computed within a class and neither one ever sees the class mix. The PR curve collapses over the same three rows, because precision is the one metric that asks "of the things I flagged, how many were real?"

At 1% prevalence and a threshold of 0.50.5, this model flags 2,079 false positives to catch 79 true ones — precision 0.0370.037. Accuracy still reads 0.7900.790 (all but unchanged from the 0.7920.792 it reported on balanced data) and the ROC curve does not move at all. Two of the three headline numbers are telling you nothing is wrong.

Python
from sklearn.metrics import roc_auc_score, average_precision_score

roc_auc = roc_auc_score(y_true, y_scores)    # AUC-ROC
pr_auc  = average_precision_score(y_true, y_scores)  # AUC-PR

Regression metrics

For regression tasks, the most common metrics are:

MAE=1n∑i=1n∣yi−y^i∣\text{MAE} = \frac{1}{n}\sum_{i=1}^n |y_i - \hat{y}_i| MSE=1n∑i=1n(yi−y^i)2,RMSE=MSE\text{MSE} = \frac{1}{n}\sum_{i=1}^n (y_i - \hat{y}_i)^2, \qquad \text{RMSE} = \sqrt{\text{MSE}} R2=1−∑(yi−y^i)2∑(yi−yˉ)2R^2 = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2}

MAE vs MSE: MSE squares the residuals, giving disproportionate weight to large errors (outliers). Use MAE when outliers are common and you don't want them to dominate. Use RMSE when large errors are especially bad.

R² measures explained variance: R²=1 is perfect fit, R²=0 means the model is no better than predicting the mean, R² < 0 is possible (worse than mean).

Multi-class metrics

For K>2K > 2 classes, average precision/recall across classes:

  • Macro average: compute metric per class, take unweighted mean — treats all classes equally regardless of size
  • Micro average: pool all TP/FP/FN across classes, then compute — dominated by frequent classes
  • Weighted average: weight each class by its support — best when class imbalance is present

For the geometric derivation of the ROC curve and the proof that AUC equals the rank-sum probability, see the deep dive:

Deep dive
ROC Curve and AUC: Derivation and Rank-Sum Interpretation
Includes runnable notebook

Exercise

A classifier has 90 true positives, 10 false positives, and 30 false negatives. What is its F1 score?

Need a hint?

Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2·P·R/(P+R).

Exercise

For a rare-disease screening test (1% prevalence), why is accuracy a misleading metric and what should you use instead?

Need a hint?

What accuracy does a classifier that always predicts 'healthy' achieve?

Exercise

You have a highly imbalanced binary dataset (0.5% positive rate). Should you use ROC-AUC or PR-AUC to compare two classifiers, and why?

Need a hint?

ROC uses true negative rate; PR does not — which one can a classifier 'game' by exploiting the large negative class?

Common mistakes

  1. Reporting only accuracy on imbalanced data: a 99% accuracy can mean the classifier does nothing useful. Always check precision/recall/F1 too.
  2. Choosing the threshold after seeing test data: threshold should be tuned on the validation set, not the test set.
  3. Conflating AUC-ROC and AUC-PR: for imbalanced data they can diverge dramatically. A model with AUC-ROC=0.95 can have AUC-PR=0.30.
  4. Using R² alone for regression: a high R² can coexist with large systematic errors. Always plot residuals.
  5. Macro averaging with very imbalanced classes: if one class has 5 samples, a bad macro-average can be dominated by noise on that class.

Key takeaways

  • The confusion matrix (TP/FP/TN/FN) is the foundation of all classification metrics.
  • Precision = "what fraction of my positives are real"; Recall = "what fraction of real positives did I find." They trade off at each threshold.
  • F1 is the harmonic mean of P and R; F-beta lets you weight one more heavily.
  • ROC-AUC measures ranking quality across all thresholds; AUC = P(positive ranked above negative).
  • PR-AUC is preferred for severe class imbalance — it ignores the huge true-negative pool that inflates ROC.
  • RMSE penalizes outliers more than MAE; R² measures explained variance relative to the mean predictor.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Classification & Regression Metri…Linear & Logistic Regress…KNN & Decision TreesNeural NetworksLogistic RegressionValidation StrategiesRank Ordering & Model Com…LLM EvaluationCalibration & UncertaintyML Systems Design & Probl…The ML Project LoopConcordance: the c-statis…ROC Curve and AUC: Deriva…Platt Scaling and Isotoni…