Gaussian ProcessesOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Gaussian Processes

Bayesian linear regression puts a distribution over weights. A Gaussian process (GP) goes one level up and puts a distribution directly over functions — without ever committing to a parametric form like "a line" or "a degree-3 polynomial." The result is a non-parametric model that fits flexible curves and, crucially, reports honest uncertainty everywhere. GPs are the gold standard when data is scarce and calibrated error bars matter.

The forecasting thread

Return to the site's running daily-demand problem from the project-loop hub. The linear-regression baseline gave a point forecast; a GP gives the same forecast with honest uncertainty — a band that stays tight where history is dense and fans out over holidays, promotions, and the days just after the regime shift. Same problem, better for a new reason: now a safety-stock decision can use the band, not just the mean.

Real-world examples: Bayesian optimization of expensive experiments and hyperparameters (next lesson), geostatistics ("kriging"), robotics and control with safety guarantees, and small-data scientific modeling.

A distribution over functions

A GP is defined so that the function values at any finite set of input points are jointly Gaussian. It's specified by a mean function m(x)m(\mathbf{x}) (usually taken to be 0) and a covariance / kernel function k(x,x′)k(\mathbf{x}, \mathbf{x}'):

f(x)∼GP(m(x),  k(x,x′))f(\mathbf{x}) \sim \mathcal{GP}\big(m(\mathbf{x}),\; k(\mathbf{x}, \mathbf{x}')\big)

The kernel is the whole model: k(x,x′)k(\mathbf{x}, \mathbf{x}') says how correlated the function's values at x\mathbf{x} and x′\mathbf{x}' are. Nearby inputs → highly correlated outputs → smooth functions.

The most common choice is the RBF (squared-exponential) kernel:

k(x,x′)=σf2exp⁡ ⁣(−∥x−x′∥22ℓ2)k(\mathbf{x}, \mathbf{x}') = \sigma_f^2 \exp\!\left(-\frac{\lVert \mathbf{x} - \mathbf{x}'\rVert^2}{2\ell^2}\right)
  • ℓ\ell — the length-scale: how far apart inputs must be before their outputs decorrelate (controls wiggliness).
  • σf2\sigma_f^2 — the signal variance (vertical scale).

The GP posterior

Condition the prior on observed data (X,y)(\mathbf{X}, \mathbf{y}) with noise variance σn2\sigma_n^2. Because everything is jointly Gaussian, the posterior at test inputs X∗\mathbf{X}_* is Gaussian with closed-form mean and covariance:

μ∗=K∗⊤(K+σn2I)−1y\boldsymbol{\mu}_* = \mathbf{K}_*^{\top}\big(\mathbf{K} + \sigma_n^2\mathbf{I}\big)^{-1}\mathbf{y} Σ∗=K∗∗−K∗⊤(K+σn2I)−1K∗\boldsymbol{\Sigma}_* = \mathbf{K}_{**} - \mathbf{K}_*^{\top}\big(\mathbf{K} + \sigma_n^2\mathbf{I}\big)^{-1}\mathbf{K}_*

where K=k(X,X)\mathbf{K} = k(\mathbf{X},\mathbf{X}), K∗=k(X,X∗)\mathbf{K}_* = k(\mathbf{X}, \mathbf{X}_*), and K∗∗=k(X∗,X∗)\mathbf{K}_{**} = k(\mathbf{X}_*, \mathbf{X}_*). The viz plots exactly this — the mean curve and the ±2σ\pm 2\sigma band:

Gaussian process regression

-4-2024
observations5
length-scale1.0
avg ±σ width0.48

Five observations pin down a distribution over functions. The blue band is the ±2σ posterior: it pinches to zero at each data point and balloons where there's no data. The length-scale controls smoothness — small values let the function wiggle to hit every point (and stay uncertain between them); large values force a smooth, confident fit.

The band pinches at the data, balloons away from it

At an observed point the posterior variance drops to ~the noise level (the model has seen it). Between and beyond observations the variance grows back toward the prior σf2\sigma_f^2. That's a GP being honest: "I'm confident here because I have data; I'm unsure there because I don't." The length-scale ℓ\ell sets how quickly confidence decays as you move away from a point.

The kernel is the prior over functions

Changing the kernel changes what functions are a priori plausible — it's how you inject assumptions:

KernelEncodesUse
RBF / squared-expvery smooth functionsdefault, smooth signals
Matérnrougher, finitely-differentiablephysical processes
Periodicrepeating structureseasonality
Linearstraight-line trendsrecovers Bayesian linear regression

Kernels compose: sum and product of kernels are kernels, so you can build "smooth + periodic + linear trend." The kernel hyperparameters (ℓ\ell, σf2\sigma_f^2, σn2\sigma_n^2) are typically fit by maximizing the marginal likelihood — the GP's built-in, automatic Occam's razor that balances fit against smoothness.

Strengths and the big limitation

  • ✅ Calibrated uncertainty everywhere, exact Bayesian inference in closed form, works with very little data, no architecture to design — just a kernel.
  • ❌ Cubic cost. Computing (K+σn2I)−1(\mathbf{K} + \sigma_n^2\mathbf{I})^{-1} is O(N3)O(N^3) in the number of training points and O(N2)O(N^2) memory. Beyond ~10⁴ points you need sparse/inducing-point approximations (e.g. SVGP) or it becomes infeasible.

GP = Bayesian linear regression with infinite features

A GP with an RBF kernel is equivalent to Bayesian linear regression using infinitely many basis functions (the kernel trick — same idea as SVMs). That's why it's non-parametric: the "weights" live in an infinite-dimensional space, but the kernel lets you work with the finite N×NN\times N matrix instead.

Exercise

In a Gaussian process, what does the kernel function k(x, x′) specify?

Need a hint?

It's what makes the GP a *prior over functions*.

Exercise

A GP posterior's ±2σ uncertainty band behaves how, as you move across the input space?

Need a hint?

Where has the model actually seen data?

Exercise

What is the main scalability limitation of exact Gaussian process regression with N training points?

Need a hint?

It involves inverting an N×N kernel matrix.

Common mistakes

  1. Treating the GP as a point predictor. The variance is the point — using only μ∗\boldsymbol{\mu}_* wastes what makes a GP valuable.
  2. Mis-set length-scale. Too small → overfits, wild uncertainty between points; too large → oversmooths and ignores real structure. Fit it by marginal likelihood, don't eyeball it.
  3. Ignoring the O(N3)O(N^3) wall. A dense GP silently fails to scale; switch to inducing-point methods for big data.
  4. Forgetting the noise term σn2\sigma_n^2. Without it the GP interpolates every point exactly (overfitting noisy data) and the kernel matrix can be singular.

Key takeaways

  • A Gaussian process is a distribution over functions: any finite set of function values is jointly Gaussian, defined by a mean and a kernel k(x,x′)k(\mathbf{x},\mathbf{x}').
  • The kernel is the prior — it sets correlation/smoothness; the length-scale ℓ\ell controls how fast confidence decays away from data.
  • The posterior mean and variance are closed-form; the ±2σ\pm2\sigma band pinches at observations and widens where there's no data.
  • GPs give calibrated uncertainty with little data but cost O(N3)O(N^3) — use inducing-point approximations at scale.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Gaussian ProcessesProbability & StatisticsLinear & Logistic Regress…Calibration & UncertaintyBayesian Linear RegressionLinear RegressionThe Kernel TrickBayesian OptimizationProbability DistributionsThe ML Project Loop