Bayesian Inference
MLE gives a single point estimate of . But what if we have prior beliefs about — for example, that weights in a neural network should be small? Bayesian inference formalizes this by treating itself as a random variable, so the output is a whole distribution over rather than one number.
Real-world examples: Bayesian neural networks model parameter uncertainty; spam filters update beliefs with each new email; medical diagnosis combines prior prevalence with test sensitivity/specificity.
Bayes' theorem
The posterior combines what the data says (likelihood) with what we believed before seeing data (prior).
| Term | Meaning |
|---|---|
| Prior | Belief about before seeing data |
| Likelihood | How probable the data is under |
| Posterior | Updated belief after seeing data |
| Evidence | Normalizing constant; often intractable |
The evidence is just whatever number makes the posterior integrate to :
Because it does not depend on , we often drop it and write the unnormalized posterior, restoring the constant only at the end:
Bayesian updating: Beta-Bernoulli
The prior (faint) encodes belief about a coin's bias θ before data. Each observed flip updates it to the posterior (bold) = Beta(α+heads, β+tails). More data → a sharper, more confident posterior.
Worked example: medical test (Bayes' theorem in numbers)
Bayes' theorem also works for discrete events. A disease affects of the population. A test is sensitive (catches sick people) and specific (correctly clears healthy people). You test positive. What is the probability you are actually sick?
Define the events: = "has the disease", = "tests positive". We are given:
We want . Apply Bayes' theorem:
Step 1: compute the evidence by summing over both ways to test positive (sick or healthy):
Step 2 — divide:
Despite a positive result from a " accurate" test, you are only about likely to be sick. The prior is so small that false positives from the huge healthy population swamp the true positives. This is the classic base-rate lesson: the prior matters enormously.
Updating beliefs with data
Bayesian inference is sequential. Start with a prior; after seeing data, your posterior becomes the prior for the next observation.
Intuition: Toss a coin once and see heads. . After 100 heads in a row, .
The posterior concentrates around the true parameter as data accumulates:
Conjugate priors
A prior is conjugate to a likelihood if the posterior is in the same family as the prior — making the update analytical (no integral to evaluate).
| Likelihood | Conjugate Prior | Posterior |
|---|---|---|
| Bernoulli | Beta | Beta |
| Gaussian known | Gaussian | Gaussian (updated mean/precision) |
| Categorical | Dirichlet | Dirichlet |
The update is derived in 5 steps: write the Bernoulli likelihood , write the Beta prior, multiply the two, collect the exponents of and , and recognize the result as another Beta kernel. No integration required — conjugacy does all the work. Starting from with heads in flips, the posterior is with mean — pulled toward 0.5 by the prior vs. the MLE of 0.700. For the complete 5-step derivation, the pseudo-count interpretation, and sequential updating animations:
MAP estimation
Maximum a Posteriori (MAP) estimation finds the mode of the posterior — the single most probable :
Because the evidence does not depend on , it cannot change where the maximum sits, so we may drop it. Taking (which is monotonic, so it preserves the location of the maximum) turns the product into a sum:
So MAP = MLE + log-prior penalty. If the prior is flat (uniform), is a constant and MAP collapses back to MLE — the data-only estimate. The prior only matters when it is informative.
L2 regularization is MAP with a Gaussian prior
Put a zero-mean Gaussian prior on each weight, . Its log is Plug this into the MAP objective (and flip the sign to turn the maximization into the usual loss minimization): Setting , that is exactly L2 (Ridge) regularization with penalty . A tighter prior (small ) means a stronger penalty (large ). L2 regularization is MAP estimation with a Gaussian prior.
Similarly, L1 regularization (Lasso) corresponds to a Laplace prior , whose log is — an absolute-value penalty .
Worked example: MAP vs MLE for the coin
The Beta-Bernoulli model gives a clean closed form for both estimates, so we can compare them by hand. With heads in flips and prior :
MLE maximizes the likelihood . Differentiate the log-likelihood and set it to zero:
MAP maximizes the posterior, i.e. the mode of . The mode of is , so:
Numbers. With , , :
The MAP estimate sits between the prior mode () and the MLE (), pulled toward the prior. Note all three summaries differ: , the posterior mode (MAP) , and the posterior mean . As , the and corrections become negligible and — the data overwhelms the prior.
alpha_prior, beta_prior = 2, 2
k, n = 7, 10
alpha_post = alpha_prior + k # 9
beta_post = beta_prior + (n - k) # 5
mle = k / n
map_est = (alpha_post - 1) / (alpha_post + beta_post - 2)
mean_est = alpha_post / (alpha_post + beta_post)
print(f"MLE: {mle:.3f}") # 0.700
print(f"MAP (mode): {map_est:.3f}") # 0.667
print(f"Posterior mean: {mean_est:.3f}") # 0.643
Bayesian vs frequentist
| Frequentist (MLE) | Bayesian | |
|---|---|---|
| is... | Fixed unknown constant | Random variable |
| Output | Point estimate | Full posterior distribution |
| Uncertainty | Confidence intervals (tricky to interpret) | Credible intervals (direct probability) |
| Prior knowledge | Cannot incorporate | Naturally incorporated |
| Scalability | Fast | Can be slow (posterior integration) |
Predictive distribution
Rather than a single , Bayesian models make predictions by averaging over all parameters weighted by the posterior:
This marginalizes out and naturally quantifies uncertainty in the prediction. In practice, this integral is often approximated with variational inference or MCMC.
When the posterior has no conjugate form, MCMC samples it directly — knowing the density only up to its normalizing constant:
Bayesian deep learning
Full Bayesian inference over neural network weights is computationally intractable. Practical approximations include: dropout as approximate Bayes (Gal & Ghahramani, 2016), variational inference (Bayes by Backprop), and deep ensembles (train multiple networks from different initializations).
Exercise
P(Disease | Positive test) is the posterior. Which theorem connects it to P(Positive | Disease)?
Need a hint?
We need to 'flip' the conditional.
Exercise
MAP estimation with a Gaussian prior N(0, τ²) on weights is equivalent to what regularization?
Need a hint?
The log of a Gaussian prior involves the squared norm of the weights.
Exercise
You use a Beta(2, 2) prior for a coin's bias p. After observing 6 heads and 4 tails, what is the posterior?
Need a hint?
For Beta-Bernoulli conjugacy: posterior = Beta(α + heads, β + tails).
Common mistakes
- Choosing an uninformative prior carelessly — a flat prior over a parameter is not uninformative after a change of variables (Jeffreys prior addresses this).
- Treating the posterior mean as the answer without acknowledging posterior width — narrow posteriors can be overconfident on small datasets.
- Confusing MAP estimation (maximizes the posterior, ignores uncertainty) with full Bayesian inference (integrates over the posterior).
Key takeaways
- Bayesian inference treats model parameters as random variables with a prior distribution.
- The posterior = likelihood × prior (normalized) — it combines data evidence with prior beliefs.
- MAP = MLE + log-prior; Gaussian prior → L2 regularization; Laplace prior → L1 regularization.
- The predictive distribution averages over parameters, naturally capturing uncertainty.
Related concepts
- Maximum Likelihood Estimation: the data-only special case
- Probability Distributions: the distributions used as priors and likelihoods
- Probabilistic Models & EM: Bayesian mixture models