Bayesian Linear Regression
Ordinary least squares returns a single best-fit line and a single prediction. But "the line is " hides a crucial question: how sure are we? Bayesian linear regression answers it by treating the weights as a probability distribution rather than a point. Instead of one line, you get a distribution over lines, and every prediction comes with a calibrated error bar that widens where you have no data. That error bar is a first-class evaluation output: the Bayesian view upgrades the loop's evaluation slot from "how good is the point prediction?" to "how much should we trust it?".
Real-world examples: any setting where overconfidence is dangerous — scientific modeling with few samples, A/B-test effect sizes, active learning (where to sample next), and as the foundation for Gaussian processes and Bayesian optimization (next lessons).
From a point estimate to a distribution
OLS finds the weights minimizing squared error — one vector. The Bayesian view instead asks for the full posterior distribution over given the data, via Bayes' rule:
- Prior — before seeing data, we believe weights are small (a Gaussian centered at 0).
- Likelihood: assume Gaussian noise: , .
- Posterior: the updated belief after the data.
The same prior → posterior updating you saw for a single parameter, now over the whole weight vector:
Bayesian updating: Beta-Bernoulli
The prior (faint) encodes belief about a coin's bias θ before data. Each observed flip updates it to the posterior (bold) = Beta(α+heads, β+tails). More data → a sharper, more confident posterior.
The posterior (closed form)
Because the prior is Gaussian and the likelihood is Gaussian, the posterior is also Gaussian (the Gaussian is conjugate to itself here) — no sampling needed:
Ridge regression is the MAP estimate
The posterior mean is exactly the ridge regression solution with . So Bayesian linear regression contains ridge: the Gaussian prior's precision is the L2 penalty. The difference is that the Bayesian version also gives you — the full covariance, i.e. the uncertainty — not just the point.
The predictive distribution
For a new input , we don't commit to one weight vector — we average over all of them, weighted by the posterior. This marginalization yields a Gaussian predictive distribution:
The predictive variance has two parts: irreducible observation noise (), and model uncertainty () that depends on where you predict.
Uncertainty grows away from the data
The model-uncertainty term is small near observed inputs and large in regions with no data — extrapolation. This is the headline benefit: a Bayesian model is honest about not knowing, instead of confidently extrapolating a single line to infinity. Active learning exploits exactly this: sample where predictive variance is highest.
Why marginalize instead of picking the best weights?
Using only the single best (MAP) throws away the spread of plausible models. Two weight vectors that fit the data equally well might disagree wildly on a far-away ; averaging over the posterior captures that disagreement as predictive variance. Marginalization is what turns "here's a guess" into "here's a guess and how much it could be wrong."
Exercise
In Bayesian linear regression with a Gaussian prior over weights and Gaussian observation noise, what is the form of the posterior over the weights?
Need a hint?
The Gaussian is conjugate to itself here.
Exercise
How does the posterior mean of Bayesian linear regression relate to ridge regression?
Need a hint?
What plays the role of the L2 penalty λ?
Exercise
The predictive variance in Bayesian linear regression is σ²(x*) = β⁻¹ + x*ᵀ S_N x*. Why does it grow for inputs far from the training data?
Need a hint?
Which term depends on where you predict?
Common mistakes
- Reporting only the posterior mean. That's just (regularized) point regression — the whole point is the variance and the predictive interval.
- Forgetting the two variance sources. Predictive variance = noise + model uncertainty; near data, only the noise floor remains.
- Thinking it's unrelated to ridge. The posterior mean is ridge; the prior precision is the L2 penalty.
- Assuming you always need sampling (MCMC). With Gaussian prior + Gaussian likelihood the posterior is closed-form — sampling is only needed for non-conjugate models.
Key takeaways
- Bayesian linear regression puts a prior over weights and computes the posterior — a distribution over lines, not a single fit.
- With Gaussian prior + Gaussian noise the posterior is closed-form; its mean equals ridge regression (prior precision = L2 penalty).
- The predictive distribution marginalizes over the posterior, giving a mean and a variance = noise + model uncertainty.
- Predictive uncertainty grows away from the data, making the model honest about extrapolation — the basis for active learning, GPs, and Bayesian optimization.
Related concepts
- Bayesian Inference: the prior → posterior machinery this lesson applies to weights
- Regularization: ridge regression, which is the MAP estimate of this model
- Gaussian Processes: the non-parametric extension: a distribution over functions, not weights