Gaussian Processes
Bayesian linear regression puts a distribution over weights. A Gaussian process (GP) goes one level up and puts a distribution directly over functions — without ever committing to a parametric form like "a line" or "a degree-3 polynomial." The result is a non-parametric model that fits flexible curves and, crucially, reports honest uncertainty everywhere. GPs are the gold standard when data is scarce and calibrated error bars matter.
The forecasting thread
Return to the site's running daily-demand problem from the project-loop hub. The linear-regression baseline gave a point forecast; a GP gives the same forecast with honest uncertainty — a band that stays tight where history is dense and fans out over holidays, promotions, and the days just after the regime shift. Same problem, better for a new reason: now a safety-stock decision can use the band, not just the mean.
Real-world examples: Bayesian optimization of expensive experiments and hyperparameters (next lesson), geostatistics ("kriging"), robotics and control with safety guarantees, and small-data scientific modeling.
A distribution over functions
A GP is defined so that the function values at any finite set of input points are jointly Gaussian. It's specified by a mean function (usually taken to be 0) and a covariance / kernel function :
The kernel is the whole model: says how correlated the function's values at and are. Nearby inputs → highly correlated outputs → smooth functions.
The most common choice is the RBF (squared-exponential) kernel:
- — the length-scale: how far apart inputs must be before their outputs decorrelate (controls wiggliness).
- — the signal variance (vertical scale).
The GP posterior
Condition the prior on observed data with noise variance . Because everything is jointly Gaussian, the posterior at test inputs is Gaussian with closed-form mean and covariance:
where , , and . The viz plots exactly this — the mean curve and the band:
Gaussian process regression
Five observations pin down a distribution over functions. The blue band is the ±2σ posterior: it pinches to zero at each data point and balloons where there's no data. The length-scale controls smoothness — small values let the function wiggle to hit every point (and stay uncertain between them); large values force a smooth, confident fit.
The band pinches at the data, balloons away from it
At an observed point the posterior variance drops to ~the noise level (the model has seen it). Between and beyond observations the variance grows back toward the prior . That's a GP being honest: "I'm confident here because I have data; I'm unsure there because I don't." The length-scale sets how quickly confidence decays as you move away from a point.
The kernel is the prior over functions
Changing the kernel changes what functions are a priori plausible — it's how you inject assumptions:
| Kernel | Encodes | Use |
|---|---|---|
| RBF / squared-exp | very smooth functions | default, smooth signals |
| Matérn | rougher, finitely-differentiable | physical processes |
| Periodic | repeating structure | seasonality |
| Linear | straight-line trends | recovers Bayesian linear regression |
Kernels compose: sum and product of kernels are kernels, so you can build "smooth + periodic + linear trend." The kernel hyperparameters (, , ) are typically fit by maximizing the marginal likelihood — the GP's built-in, automatic Occam's razor that balances fit against smoothness.
Strengths and the big limitation
- ✅ Calibrated uncertainty everywhere, exact Bayesian inference in closed form, works with very little data, no architecture to design — just a kernel.
- ❌ Cubic cost. Computing is in the number of training points and memory. Beyond ~10⁴ points you need sparse/inducing-point approximations (e.g. SVGP) or it becomes infeasible.
GP = Bayesian linear regression with infinite features
A GP with an RBF kernel is equivalent to Bayesian linear regression using infinitely many basis functions (the kernel trick — same idea as SVMs). That's why it's non-parametric: the "weights" live in an infinite-dimensional space, but the kernel lets you work with the finite matrix instead.
Exercise
In a Gaussian process, what does the kernel function k(x, x′) specify?
Need a hint?
It's what makes the GP a *prior over functions*.
Exercise
A GP posterior's ±2σ uncertainty band behaves how, as you move across the input space?
Need a hint?
Where has the model actually seen data?
Exercise
What is the main scalability limitation of exact Gaussian process regression with N training points?
Need a hint?
It involves inverting an N×N kernel matrix.
Common mistakes
- Treating the GP as a point predictor. The variance is the point — using only wastes what makes a GP valuable.
- Mis-set length-scale. Too small → overfits, wild uncertainty between points; too large → oversmooths and ignores real structure. Fit it by marginal likelihood, don't eyeball it.
- Ignoring the wall. A dense GP silently fails to scale; switch to inducing-point methods for big data.
- Forgetting the noise term . Without it the GP interpolates every point exactly (overfitting noisy data) and the kernel matrix can be singular.
Key takeaways
- A Gaussian process is a distribution over functions: any finite set of function values is jointly Gaussian, defined by a mean and a kernel .
- The kernel is the prior — it sets correlation/smoothness; the length-scale controls how fast confidence decays away from data.
- The posterior mean and variance are closed-form; the band pinches at observations and widens where there's no data.
- GPs give calibrated uncertainty with little data but cost — use inducing-point approximations at scale.
Related concepts
- Bayesian Linear Regression: the parametric special case; a GP is its infinite-feature limit
- Kernel Trick: the same "work via a kernel instead of explicit features" idea
- Bayesian Optimization: using a GP surrogate to optimize expensive black-box functions
- Probability Distributions: the multivariate Gaussian a GP is built from