Bias-Variance Decomposition
Every supervised model faces the same trade-off: the more flexible it is, the more it chases training-set noise; the simpler it is, the more it ignores real signal. The bias-variance decomposition quantifies this formally, splitting expected test error into three additive components that cannot be reduced independently.
Setup and notation
Let the true relationship be , where is zero-mean noise with and . We fit a model on a training set drawn from the same distribution. The expected test error at a fixed point (expectation over many random draws of ) is:
The algebraic decomposition
Let be the mean prediction across training sets. Add and subtract :
Square and expand. Cross-terms involving vanish because is independent of and . The cross-term between bias and variation vanishes because by definition of :
The three terms:
- Bias — the error your model makes on average; structural, not fixable with more data if the model family is wrong.
- Variance — how much the prediction swings when you redraw the training set; cured by regularization, ensembling, or more data.
- — label noise baked into the targets; no model can escape it.
The tradeoff in KNN
For -nearest neighbors, shifts both terms in opposite directions:
| Prediction | copy the nearest point | global majority vote |
| Bias | minimal | maximal: predicts the mean |
| Variance | huge: one noisy neighbor flips the answer | tiny |
| Boundary | jagged islands around noise points | nearly flat |
Total test error traces a U-shape as grows: Bias² rises monotonically while Variance falls. The optimal sits at the bottom of the U, found by cross-validation.
Estimating Bias² and Variance empirically
Run the model on many resamples and measure the spread vs. the average error:
- Draw training sets from the data distribution.
- Fit ; record predictions at a fixed test point .
- — the empirical mean prediction.
- .
- .
import numpy as np
from sklearn.neighbors import KNeighborsRegressor
rng = np.random.default_rng(42)
n_train, M = 50, 500
x_test = np.array([[0.5]])
def f_true(x): return np.sin(2 * np.pi * x).ravel()
print(f"{'k':>3} {'Bias²':>8} {'Var':>8} {'Total':>8}")
for k in [1, 3, 5, 10, 20, 50]:
preds = []
for _ in range(M):
X = rng.uniform(0, 1, (n_train, 1))
y = f_true(X) + rng.normal(0, 0.2, n_train)
preds.append(KNeighborsRegressor(n_neighbors=k).fit(X, y).predict(x_test)[0])
preds = np.array(preds)
bias2 = (preds.mean() - f_true(x_test)[0]) ** 2
variance = preds.var()
print(f"{k:>3} {bias2:>8.4f} {variance:>8.4f} {bias2+variance:>8.4f}")
Diagnosing a bias or variance problem
Training vs. validation error
Compare training error and validation error. Both high → bias problem: the model cannot fit even what it has seen — add capacity, more features, or reduce regularization. Training low, validation high → variance problem: regularize, simplify, get more data, or use ensembles. More data cures variance; it never cures bias.
| Symptom | Cause | Fix |
|---|---|---|
| High train error + high val error | High bias | More capacity, fewer constraints |
| Low train error + high val error | High variance | Regularize, ensemble, more data |
| Errors decrease with more data | Variance-limited | Collect more data |
| Errors plateau with more data | Bias-limited | Change model family |
The modern footnote on double descent
Classical theory predicts a U-shaped test error curve. Very large neural networks add a twist: past the interpolation threshold (the point where the model can perfectly memorize training data) test error can fall again as the model grows further. Among the many zero-training-error solutions, gradient descent tends to find the smoothest one, which behaves as if implicitly regularized. The bias–variance lens is still the right starting intuition; just know the edges are an active research area.
Related concepts
- The Bias–Variance Tradeoff: the full lesson with interactive KNN visualization
- K-Nearest Neighbors: the model used to illustrate the tradeoff
- Regularization: Ridge & Lasso: one dial for reducing variance
- Bagging and Random Forests: ensembles that cancel variance while preserving low bias