The Bias–Variance TradeoffOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

The Bias–Variance Tradeoff

You've now met two models with an obvious "complexity dial": the kk in K-Nearest Neighbors and the depth of a decision tree. Turn the dial one way and the model ignores real structure; turn it the other way and it hallucinates structure that isn't there. This tension has a name (the bias–variance tradeoff) and it governs every model you will ever train. It is the theory underneath the loop's evaluation stage: it explains why a model that looks great on training data can fail on new data, and what the generalization gap you measure is made of.

The decomposition

Expected test error splits into three additive pieces: Bias2+Variance+σ2\text{Bias}^2 + \text{Variance} + \sigma^2 (irreducible noise). Bias is how far the average prediction is from the truth; variance is how much it swings across different training sets. Total error is minimized somewhere in between — simpler models accept bias to cut variance, complex models do the reverse.

Deep dive
Bias-Variance Decomposition
Includes runnable notebook

Seeing it in k

For KNN, kk controls how much the prediction averages over:

k=1k = 1k=nk = n
Predictioncopy the nearest pointaverage everyone
Biasminimalmaximal: predicts the global mean
Variancehuge: one noisy neighbor flips the answertiny
Boundaryjagged islands around noise pointsnearly flat

Small kk = low bias, high variance. Large kk = high bias, low variance. The best kk (found by cross-validation) sits where the two curves cross.

The definitions above talk about "many training sets" — an experiment you can actually run. Below, the same k-NN model is fit to 24 different resamples of the same noisy curve. Watch what the k slider does to the spread of fits (variance) versus the accuracy of their average (bias):

One model, many training sets

x (input)y
— individual fits (24 resamples)— average fit- - true function
bias²0.006
variance0.133
bias² + variance0.138

Each thin rose curve is a k-NN regressor fit to a fresh resample of the same noisy sine data. Their spread is variance; the gap between their average (teal) and the truth (dashed) is bias. Small k: the cloud is wild but centered on the truth. Large k: the cloud collapses, but onto the wrong, over-smoothed curve.

Diagnosing which problem you have

Compare training and validation error. Both high → bias problem: your model can't even fit the data it has seen — add capacity or features. Training low, validation high → variance problem: regularize, simplify, or get more data. More data fixes variance; it never fixes bias.

Exercise

Training error and validation error are both high, and close together. What's the diagnosis and cure?

Need a hint?

The model can't even fit the data it has seen.

Key takeaways

  • Test error = bias² + variance + irreducible noise
  • Small kk / deep trees → low bias, high variance; large kk / shallow trees → the reverse
  • Diagnose by comparing training vs. validation error; the gap is variance, the floor is bias
  • More data cures variance, never bias; ensembles cancel the variance of low-bias models

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
The Bias–Variance TradeoffLinear & Logistic Regress…Regularization: Ridge & L…K-Nearest NeighborsDecision TreesLearning Theory & General…Bias-Variance Decompositi…VC Dimension