Bias-Variance DecompositionOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Bias-Variance Decomposition

Every supervised model faces the same trade-off: the more flexible it is, the more it chases training-set noise; the simpler it is, the more it ignores real signal. The bias-variance decomposition quantifies this formally, splitting expected test error into three additive components that cannot be reduced independently.

Setup and notation

Let the true relationship be y=f(x)+εy = f(\mathbf{x}) + \varepsilon, where ε\varepsilon is zero-mean noise with E[ε]=0\mathbb{E}[\varepsilon] = 0 and Var(ε)=σ2\text{Var}(\varepsilon) = \sigma^2. We fit a model f^\hat{f} on a training set D\mathcal{D} drawn from the same distribution. The expected test error at a fixed point x\mathbf{x} (expectation over many random draws of D\mathcal{D}) is:

ED ⁣[(y−f^(x))2]\mathbb{E}_\mathcal{D}\!\left[(y - \hat{f}(\mathbf{x}))^2\right]

The algebraic decomposition

Let μ(x)=ED[f^(x)]\mu(\mathbf{x}) = \mathbb{E}_\mathcal{D}[\hat{f}(\mathbf{x})] be the mean prediction across training sets. Add and subtract μ\mu:

y−f^=(f(x)−μ(x))⏟bias+(μ(x)−f^(x))⏟variation+ε⏟noisey - \hat{f} = \underbrace{(f(\mathbf{x}) - \mu(\mathbf{x}))}_{\text{bias}} + \underbrace{(\mu(\mathbf{x}) - \hat{f}(\mathbf{x}))}_{\text{variation}} + \underbrace{\varepsilon}_{\text{noise}}

Square and expand. Cross-terms involving ε\varepsilon vanish because ε\varepsilon is independent of f^\hat{f} and E[ε]=0\mathbb{E}[\varepsilon] = 0. The cross-term between bias and variation vanishes because ED[μ−f^]=0\mathbb{E}_\mathcal{D}[\mu - \hat{f}] = 0 by definition of μ\mu:

E ⁣[(y−f^)2]=(f(x)−μ(x))2⏟Bias2(f^)+E ⁣[(f^(x)−μ(x))2]⏟Variance(f^)+σ2⏟irreducible noise\mathbb{E}\!\left[(y - \hat{f})^2\right] = \underbrace{(f(\mathbf{x}) - \mu(\mathbf{x}))^2}_{\text{Bias}^2(\hat{f})} + \underbrace{\mathbb{E}\!\left[(\hat{f}(\mathbf{x}) - \mu(\mathbf{x}))^2\right]}_{\text{Variance}(\hat{f})} + \underbrace{\sigma^2}_{\text{irreducible noise}}

The three terms:

  • Bias =f(x)−ED[f^(x)]= f(\mathbf{x}) - \mathbb{E}_\mathcal{D}[\hat{f}(\mathbf{x})] — the error your model makes on average; structural, not fixable with more data if the model family is wrong.
  • Variance =ED ⁣[(f^(x)−E[f^(x)])2]= \mathbb{E}_\mathcal{D}\!\left[(\hat{f}(\mathbf{x}) - \mathbb{E}[\hat{f}(\mathbf{x})])^2\right] — how much the prediction swings when you redraw the training set; cured by regularization, ensembling, or more data.
  • σ2\sigma^2 — label noise baked into the targets; no model can escape it.

The tradeoff in KNN

For kk-nearest neighbors, kk shifts both terms in opposite directions:

k=1k = 1k=nk = n
Predictioncopy the nearest pointglobal majority vote
Biasminimalmaximal: predicts the mean
Variancehuge: one noisy neighbor flips the answertiny
Boundaryjagged islands around noise pointsnearly flat

Total test error traces a U-shape as kk grows: Bias² rises monotonically while Variance falls. The optimal kk sits at the bottom of the U, found by cross-validation.

Estimating Bias² and Variance empirically

Run the model on many resamples and measure the spread vs. the average error:

  1. Draw MM training sets D1,…,DM\mathcal{D}_1, \ldots, \mathcal{D}_M from the data distribution.
  2. Fit f^1,…,f^M\hat{f}_1, \ldots, \hat{f}_M; record predictions y^m=f^m(x0)\hat{y}_m = \hat{f}_m(\mathbf{x}_0) at a fixed test point x0\mathbf{x}_0.
  3. yˉ=1M∑my^m\bar{y} = \frac{1}{M}\sum_m \hat{y}_m — the empirical mean prediction.
  4. Variance^=1M∑m(y^m−yˉ)2\widehat{\text{Variance}} = \frac{1}{M}\sum_m (\hat{y}_m - \bar{y})^2.
  5. Bias2^=(yˉ−f(x0))2\widehat{\text{Bias}^2} = (\bar{y} - f(\mathbf{x}_0))^2.
Python
import numpy as np
from sklearn.neighbors import KNeighborsRegressor

rng = np.random.default_rng(42)
n_train, M = 50, 500
x_test = np.array([[0.5]])

def f_true(x): return np.sin(2 * np.pi * x).ravel()

print(f"{'k':>3}  {'Bias²':>8}  {'Var':>8}  {'Total':>8}")

for k in [1, 3, 5, 10, 20, 50]:
    preds = []
    for _ in range(M):
        X = rng.uniform(0, 1, (n_train, 1))
        y = f_true(X) + rng.normal(0, 0.2, n_train)
        preds.append(KNeighborsRegressor(n_neighbors=k).fit(X, y).predict(x_test)[0])
    preds = np.array(preds)
    bias2 = (preds.mean() - f_true(x_test)[0]) ** 2
    variance = preds.var()
    print(f"{k:>3}  {bias2:>8.4f}  {variance:>8.4f}  {bias2+variance:>8.4f}")

Diagnosing a bias or variance problem

Training vs. validation error

Compare training error and validation error. Both high → bias problem: the model cannot fit even what it has seen — add capacity, more features, or reduce regularization. Training low, validation high → variance problem: regularize, simplify, get more data, or use ensembles. More data cures variance; it never cures bias.

SymptomCauseFix
High train error + high val errorHigh biasMore capacity, fewer constraints
Low train error + high val errorHigh varianceRegularize, ensemble, more data
Errors decrease with more dataVariance-limitedCollect more data
Errors plateau with more dataBias-limitedChange model family

The modern footnote on double descent

Classical theory predicts a U-shaped test error curve. Very large neural networks add a twist: past the interpolation threshold (the point where the model can perfectly memorize training data) test error can fall again as the model grows further. Among the many zero-training-error solutions, gradient descent tends to find the smoothest one, which behaves as if implicitly regularized. The bias–variance lens is still the right starting intuition; just know the edges are an active research area.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Bias-Variance DecompositionInductive Bias & Equivari…The ML Project LoopVC DimensionWalk-Forward ValidationThe Bias–Variance TradeoffCalibration & UncertaintyK-Nearest NeighborsRegularization: Ridge & L…Bagging & Random Forests