The Bias–Variance Tradeoff
You've now met two models with an obvious "complexity dial": the in K-Nearest Neighbors and the depth of a decision tree. Turn the dial one way and the model ignores real structure; turn it the other way and it hallucinates structure that isn't there. This tension has a name (the bias–variance tradeoff) and it governs every model you will ever train. It is the theory underneath the loop's evaluation stage: it explains why a model that looks great on training data can fail on new data, and what the generalization gap you measure is made of.
The decomposition
Expected test error splits into three additive pieces: (irreducible noise). Bias is how far the average prediction is from the truth; variance is how much it swings across different training sets. Total error is minimized somewhere in between — simpler models accept bias to cut variance, complex models do the reverse.
Seeing it in k
For KNN, controls how much the prediction averages over:
| Prediction | copy the nearest point | average everyone |
| Bias | minimal | maximal: predicts the global mean |
| Variance | huge: one noisy neighbor flips the answer | tiny |
| Boundary | jagged islands around noise points | nearly flat |
Small = low bias, high variance. Large = high bias, low variance. The best (found by cross-validation) sits where the two curves cross.
The definitions above talk about "many training sets" — an experiment you can actually run. Below, the same k-NN model is fit to 24 different resamples of the same noisy curve. Watch what the k slider does to the spread of fits (variance) versus the accuracy of their average (bias):
One model, many training sets
Each thin rose curve is a k-NN regressor fit to a fresh resample of the same noisy sine data. Their spread is variance; the gap between their average (teal) and the truth (dashed) is bias. Small k: the cloud is wild but centered on the truth. Large k: the cloud collapses, but onto the wrong, over-smoothed curve.
Diagnosing which problem you have
Compare training and validation error. Both high → bias problem: your model can't even fit the data it has seen — add capacity or features. Training low, validation high → variance problem: regularize, simplify, or get more data. More data fixes variance; it never fixes bias.
Exercise
Training error and validation error are both high, and close together. What's the diagnosis and cure?
Need a hint?
The model can't even fit the data it has seen.
Key takeaways
- Test error = bias² + variance + irreducible noise
- Small / deep trees → low bias, high variance; large / shallow trees → the reverse
- Diagnose by comparing training vs. validation error; the gap is variance, the floor is bias
- More data cures variance, never bias; ensembles cancel the variance of low-bias models
Related concepts
- Bias–Variance Decomposition: the full derivation of test error = bias² + variance + noise
- VC Dimension: the learning-theory measure of model capacity behind the bias–variance dial