Bayesian Linear RegressionOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Bayesian Linear Regression

Ordinary least squares returns a single best-fit line and a single prediction. But "the line is y=2.1x+0.4y = 2.1x + 0.4" hides a crucial question: how sure are we? Bayesian linear regression answers it by treating the weights as a probability distribution rather than a point. Instead of one line, you get a distribution over lines, and every prediction comes with a calibrated error bar that widens where you have no data. That error bar is a first-class evaluation output: the Bayesian view upgrades the loop's evaluation slot from "how good is the point prediction?" to "how much should we trust it?".

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: any setting where overconfidence is dangerous — scientific modeling with few samples, A/B-test effect sizes, active learning (where to sample next), and as the foundation for Gaussian processes and Bayesian optimization (next lessons).

From a point estimate to a distribution

OLS finds the weights w\mathbf{w} minimizing squared error — one vector. The Bayesian view instead asks for the full posterior distribution over w\mathbf{w} given the data, via Bayes' rule:

p(w∣X,y)⏟posterior  ∝  p(y∣X,w)⏟likelihood  p(w)⏟prior\underbrace{p(\mathbf{w}\mid \mathbf{X}, \mathbf{y})}_{\text{posterior}} \;\propto\; \underbrace{p(\mathbf{y}\mid \mathbf{X}, \mathbf{w})}_{\text{likelihood}}\;\underbrace{p(\mathbf{w})}_{\text{prior}}
  • Prior p(w)=N(0,α−1I)p(\mathbf{w}) = \mathcal{N}(\mathbf{0}, \alpha^{-1}\mathbf{I}) — before seeing data, we believe weights are small (a Gaussian centered at 0).
  • Likelihood: assume Gaussian noise: y=w⊤x+ϵy = \mathbf{w}^{\top}\mathbf{x} + \epsilon, ϵ∼N(0,β−1)\epsilon \sim \mathcal{N}(0, \beta^{-1}).
  • Posterior: the updated belief after the data.

The same prior → posterior updating you saw for a single parameter, now over the whole weight vector:

Bayesian updating: Beta-Bernoulli

bias θ (0 → 1)density
posteriorBeta(2.0, 2.0)
post. mean0.500
MAP0.500

The prior (faint) encodes belief about a coin's bias θ before data. Each observed flip updates it to the posterior (bold) = Beta(α+heads, β+tails). More data → a sharper, more confident posterior.

The posterior (closed form)

Because the prior is Gaussian and the likelihood is Gaussian, the posterior is also Gaussian (the Gaussian is conjugate to itself here) — no sampling needed:

p(w∣X,y)=N(w∣mN,SN)p(\mathbf{w}\mid \mathbf{X}, \mathbf{y}) = \mathcal{N}(\mathbf{w} \mid \mathbf{m}_N, \mathbf{S}_N) SN−1=αI+β X⊤X,mN=β SN X⊤y\mathbf{S}_N^{-1} = \alpha\mathbf{I} + \beta\,\mathbf{X}^{\top}\mathbf{X}, \qquad \mathbf{m}_N = \beta\,\mathbf{S}_N\,\mathbf{X}^{\top}\mathbf{y}

Ridge regression is the MAP estimate

The posterior mean mN\mathbf{m}_N is exactly the ridge regression solution with λ=α/β\lambda = \alpha/\beta. So Bayesian linear regression contains ridge: the Gaussian prior's precision α\alpha is the L2 penalty. The difference is that the Bayesian version also gives you SN\mathbf{S}_N — the full covariance, i.e. the uncertainty — not just the point.

The predictive distribution

For a new input x∗\mathbf{x}_*, we don't commit to one weight vector — we average over all of them, weighted by the posterior. This marginalization yields a Gaussian predictive distribution:

p(y∗∣x∗,X,y)=N(y∗∣mN⊤x∗,  σN2(x∗))p(y_* \mid \mathbf{x}_*, \mathbf{X}, \mathbf{y}) = \mathcal{N}\big(y_* \mid \mathbf{m}_N^{\top}\mathbf{x}_*,\; \sigma_N^2(\mathbf{x}_*)\big) σN2(x∗)=β−1⏟noise+x∗⊤SN x∗⏟model uncertainty\sigma_N^2(\mathbf{x}_*) = \underbrace{\beta^{-1}}_{\text{noise}} + \underbrace{\mathbf{x}_*^{\top}\mathbf{S}_N\,\mathbf{x}_*}_{\text{model uncertainty}}

The predictive variance has two parts: irreducible observation noise (β−1\beta^{-1}), and model uncertainty (x∗⊤SNx∗\mathbf{x}_*^{\top}\mathbf{S}_N\mathbf{x}_*) that depends on where you predict.

Uncertainty grows away from the data

The model-uncertainty term is small near observed inputs and large in regions with no data — extrapolation. This is the headline benefit: a Bayesian model is honest about not knowing, instead of confidently extrapolating a single line to infinity. Active learning exploits exactly this: sample where predictive variance is highest.

Why marginalize instead of picking the best weights?

Using only the single best w\mathbf{w} (MAP) throws away the spread of plausible models. Two weight vectors that fit the data equally well might disagree wildly on a far-away x∗\mathbf{x}_*; averaging over the posterior captures that disagreement as predictive variance. Marginalization is what turns "here's a guess" into "here's a guess and how much it could be wrong."

Exercise

In Bayesian linear regression with a Gaussian prior over weights and Gaussian observation noise, what is the form of the posterior over the weights?

Need a hint?

The Gaussian is conjugate to itself here.

Exercise

How does the posterior mean of Bayesian linear regression relate to ridge regression?

Need a hint?

What plays the role of the L2 penalty λ?

Exercise

The predictive variance in Bayesian linear regression is σ²(x*) = β⁻¹ + x*ᵀ S_N x*. Why does it grow for inputs far from the training data?

Need a hint?

Which term depends on where you predict?

Common mistakes

  1. Reporting only the posterior mean. That's just (regularized) point regression — the whole point is the variance SN\mathbf{S}_N and the predictive interval.
  2. Forgetting the two variance sources. Predictive variance = noise + model uncertainty; near data, only the noise floor remains.
  3. Thinking it's unrelated to ridge. The posterior mean is ridge; the prior precision is the L2 penalty.
  4. Assuming you always need sampling (MCMC). With Gaussian prior + Gaussian likelihood the posterior is closed-form — sampling is only needed for non-conjugate models.

Key takeaways

  • Bayesian linear regression puts a prior over weights and computes the posterior N(mN,SN)\mathcal{N}(\mathbf{m}_N, \mathbf{S}_N) — a distribution over lines, not a single fit.
  • With Gaussian prior + Gaussian noise the posterior is closed-form; its mean equals ridge regression (prior precision = L2 penalty).
  • The predictive distribution marginalizes over the posterior, giving a mean and a variance = noise + model uncertainty.
  • Predictive uncertainty grows away from the data, making the model honest about extrapolation — the basis for active learning, GPs, and Bayesian optimization.

Related concepts

  • Bayesian Inference: the prior → posterior machinery this lesson applies to weights
  • Regularization: ridge regression, which is the MAP estimate of this model
  • Gaussian Processes: the non-parametric extension: a distribution over functions, not weights

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Bayesian Linear RegressionProbability & StatisticsLinear & Logistic Regress…Bayesian InferenceRegularization: Ridge & L…Gaussian ProcessesBayesian OptimizationThe ML Project LoopMCMC Sampling: Metropolis…