Bayesian InferenceOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Bayesian Inference

MLE gives a single point estimate of θ\theta. But what if we have prior beliefs about θ\theta — for example, that weights in a neural network should be small? Bayesian inference formalizes this by treating θ\theta itself as a random variable, so the output is a whole distribution over θ\theta rather than one number.

Real-world examples: Bayesian neural networks model parameter uncertainty; spam filters update beliefs with each new email; medical diagnosis combines prior prevalence with test sensitivity/specificity.

Bayes' theorem

p(θ∣D)⏟posterior=p(D∣θ)⏟likelihood⋅p(θ)⏟priorp(D)⏟evidence\underbrace{p(\theta | \mathcal{D})}_{\text{posterior}} = \frac{\underbrace{p(\mathcal{D} | \theta)}_{\text{likelihood}} \cdot \underbrace{p(\theta)}_{\text{prior}}}{\underbrace{p(\mathcal{D})}_{\text{evidence}}}

The posterior p(θ∣D)p(\theta | \mathcal{D}) combines what the data says (likelihood) with what we believed before seeing data (prior).

TermMeaning
Prior p(θ)p(\theta)Belief about θ\theta before seeing data
Likelihood p(D∣θ)p(\mathcal{D} \mid \theta)How probable the data is under θ\theta
Posterior p(θ∣D)p(\theta \mid \mathcal{D})Updated belief after seeing data
Evidence p(D)p(\mathcal{D})Normalizing constant; often intractable

The evidence is just whatever number makes the posterior integrate to 11:

p(D)=∫p(D∣θ) p(θ) dθp(\mathcal{D}) = \int p(\mathcal{D} | \theta)\, p(\theta)\, d\theta

Because it does not depend on θ\theta, we often drop it and write the unnormalized posterior, restoring the constant only at the end:

p(θ∣D)  ∝  p(D∣θ) p(θ)p(\theta | \mathcal{D}) \;\propto\; p(\mathcal{D} | \theta)\, p(\theta)

Bayesian updating: Beta-Bernoulli

bias θ (0 → 1)density
posteriorBeta(2.0, 2.0)
post. mean0.500
MAP0.500

The prior (faint) encodes belief about a coin's bias θ before data. Each observed flip updates it to the posterior (bold) = Beta(α+heads, β+tails). More data → a sharper, more confident posterior.

Worked example: medical test (Bayes' theorem in numbers)

Bayes' theorem also works for discrete events. A disease affects 1%1\% of the population. A test is 99%99\% sensitive (catches sick people) and 95%95\% specific (correctly clears healthy people). You test positive. What is the probability you are actually sick?

Define the events: DD = "has the disease", T+T^+ = "tests positive". We are given:

p(D)=0.01,p(T+∣D)=0.99,p(T+∣¬D)=1−0.95=0.05p(D) = 0.01, \qquad p(T^+ | D) = 0.99, \qquad p(T^+ | \neg D) = 1 - 0.95 = 0.05

We want p(D∣T+)p(D | T^+). Apply Bayes' theorem:

p(D∣T+)=p(T+∣D) p(D)p(T+)p(D | T^+) = \frac{p(T^+ | D)\, p(D)}{p(T^+)}

Step 1: compute the evidence p(T+)p(T^+) by summing over both ways to test positive (sick or healthy):

p(T+)=p(T+∣D) p(D)⏟true positive+p(T+∣¬D) p(¬D)⏟false positivep(T^+) = \underbrace{p(T^+ | D)\,p(D)}_{\text{true positive}} + \underbrace{p(T^+ | \neg D)\,p(\neg D)}_{\text{false positive}} p(T+)=(0.99)(0.01)+(0.05)(0.99)=0.0099+0.0495=0.0594p(T^+) = (0.99)(0.01) + (0.05)(0.99) = 0.0099 + 0.0495 = 0.0594

Step 2 — divide:

p(D∣T+)=0.00990.0594≈0.167p(D | T^+) = \frac{0.0099}{0.0594} \approx 0.167

Despite a positive result from a "99%99\% accurate" test, you are only about 17%17\% likely to be sick. The prior p(D)=0.01p(D) = 0.01 is so small that false positives from the huge healthy population swamp the true positives. This is the classic base-rate lesson: the prior matters enormously.

Updating beliefs with data

Bayesian inference is sequential. Start with a prior; after seeing data, your posterior becomes the prior for the next observation.

Intuition: Toss a coin once and see heads. p(fair)=0.5p(\text{fair}) = 0.5. After 100 heads in a row, p(fair)≈0p(\text{fair}) \approx 0.

The posterior concentrates around the true parameter as data accumulates:

p(θ∣x1,…,xn)∝p(θ)∏i=1np(xi∣θ)p(\theta | x_1, \ldots, x_n) \propto p(\theta) \prod_{i=1}^n p(x_i | \theta)

Conjugate priors

A prior is conjugate to a likelihood if the posterior is in the same family as the prior — making the update analytical (no integral to evaluate).

LikelihoodConjugate PriorPosterior
Bernoulli(p)(p)Beta(α,β)(\alpha, \beta)Beta(α+n1,β+n0)(\alpha + n_1, \beta + n_0)
Gaussian(μ,σ2)(\mu, \sigma^2) known σ2\sigma^2Gaussian(μ0,τ2)(\mu_0, \tau^2)Gaussian (updated mean/precision)
Categorical(π)(\boldsymbol{\pi})Dirichlet(α)(\boldsymbol{\alpha})Dirichlet(α+n)(\boldsymbol{\alpha} + \mathbf{n})

The update is derived in 5 steps: write the Bernoulli likelihood pk(1−p)n−kp^k(1-p)^{n-k}, write the Beta prior, multiply the two, collect the exponents of pp and (1−p)(1-p), and recognize the result as another Beta kernel. No integration required — conjugacy does all the work. Starting from Beta(2,2)\text{Beta}(2,2) with k=7k=7 heads in n=10n=10 flips, the posterior is Beta(9,5)\text{Beta}(9,5) with mean 9/14≈0.6439/14 \approx 0.643 — pulled toward 0.5 by the prior vs. the MLE of 0.700. For the complete 5-step derivation, the pseudo-count interpretation, and sequential updating animations:

Deep dive
Beta-Bernoulli Conjugate Posterior
Includes runnable notebook

MAP estimation

Maximum a Posteriori (MAP) estimation finds the mode of the posterior — the single most probable θ\theta:

θ^MAP=arg⁡max⁡θp(θ∣D)\hat{\theta}_\text{MAP} = \arg\max_\theta p(\theta | \mathcal{D})

Because the evidence p(D)p(\mathcal{D}) does not depend on θ\theta, it cannot change where the maximum sits, so we may drop it. Taking log⁡\log (which is monotonic, so it preserves the location of the maximum) turns the product into a sum:

θ^MAP=arg⁡max⁡θ[ p(D∣θ) p(θ) ]=arg⁡max⁡θ[ln⁡p(D∣θ)⏟MLE objective+ln⁡p(θ)⏟prior penalty]\hat{\theta}_\text{MAP} = \arg\max_\theta \big[\, p(\mathcal{D}|\theta)\,p(\theta)\,\big] = \arg\max_\theta \left[\underbrace{\ln p(\mathcal{D} | \theta)}_{\text{MLE objective}} + \underbrace{\ln p(\theta)}_{\text{prior penalty}}\right]

So MAP = MLE + log-prior penalty. If the prior is flat (uniform), ln⁡p(θ)\ln p(\theta) is a constant and MAP collapses back to MLE — the data-only estimate. The prior only matters when it is informative.

L2 regularization is MAP with a Gaussian prior

Put a zero-mean Gaussian prior on each weight, θ∼N(0,τ2)\theta \sim \mathcal{N}(0, \tau^2). Its log is ln⁡p(θ)=−∥θ∥22τ2+const.\ln p(\theta) = -\frac{\|\theta\|^2}{2\tau^2} + \text{const}. Plug this into the MAP objective (and flip the sign to turn the maximization into the usual loss minimization): θ^MAP=arg⁡min⁡θ[−ln⁡p(D∣θ)⏟data loss+12τ2∥θ∥2⏟penalty].\hat{\theta}_\text{MAP} = \arg\min_\theta \Big[\underbrace{-\ln p(\mathcal{D}|\theta)}_{\text{data loss}} + \underbrace{\tfrac{1}{2\tau^2}\|\theta\|^2}_{\text{penalty}}\Big]. Setting λ=12τ2\lambda = \tfrac{1}{2\tau^2}, that is exactly L2 (Ridge) regularization with penalty λ∥θ∥2\lambda\|\theta\|^2. A tighter prior (small τ2\tau^2) means a stronger penalty (large λ\lambda). L2 regularization is MAP estimation with a Gaussian prior.

Similarly, L1 regularization (Lasso) corresponds to a Laplace prior θ∼Laplace(0,b)\theta \sim \text{Laplace}(0, b), whose log is −1b∣θ∣+const-\frac{1}{b}|\theta| + \text{const} — an absolute-value penalty λ∥θ∥1\lambda\|\theta\|_1.

Worked example: MAP vs MLE for the coin

The Beta-Bernoulli model gives a clean closed form for both estimates, so we can compare them by hand. With kk heads in nn flips and prior Beta(α,β)\text{Beta}(\alpha, \beta):

MLE maximizes the likelihood pk(1−p)n−kp^k(1-p)^{n-k}. Differentiate the log-likelihood and set it to zero:

ddp[kln⁡p+(n−k)ln⁡(1−p)]=kp−n−k1−p=0  ⟹  p^MLE=kn\frac{d}{dp}\big[k\ln p + (n-k)\ln(1-p)\big] = \frac{k}{p} - \frac{n-k}{1-p} = 0 \;\Longrightarrow\; \hat{p}_\text{MLE} = \frac{k}{n}

MAP maximizes the posterior, i.e. the mode of Beta(α+k, β+n−k)\text{Beta}(\alpha + k,\ \beta + n - k). The mode of Beta(a,b)\text{Beta}(a, b) is a−1a+b−2\frac{a-1}{a+b-2}, so:

p^MAP=(α+k)−1(α+k)+(β+n−k)−2=α+k−1α+β+n−2\hat{p}_\text{MAP} = \frac{(\alpha + k) - 1}{(\alpha + k) + (\beta + n - k) - 2} = \frac{\alpha + k - 1}{\alpha + \beta + n - 2}

Numbers. With α=β=2\alpha = \beta = 2, k=7k = 7, n=10n = 10:

p^MLE=710=0.700,p^MAP=2+7−12+2+10−2=812≈0.667\hat{p}_\text{MLE} = \frac{7}{10} = 0.700, \qquad \hat{p}_\text{MAP} = \frac{2 + 7 - 1}{2 + 2 + 10 - 2} = \frac{8}{12} \approx 0.667

The MAP estimate sits between the prior mode (0.50.5) and the MLE (0.70.7), pulled toward the prior. Note all three summaries differ: p^MLE=0.700\hat{p}_\text{MLE} = 0.700, the posterior mode (MAP) =0.667= 0.667, and the posterior mean =0.643= 0.643. As n→∞n \to \infty, the −1-1 and −2-2 corrections become negligible and p^MAP→p^MLE\hat{p}_\text{MAP} \to \hat{p}_\text{MLE} — the data overwhelms the prior.

Python
alpha_prior, beta_prior = 2, 2
k, n = 7, 10

alpha_post = alpha_prior + k          # 9
beta_post  = beta_prior  + (n - k)    # 5

mle = k / n
map_est  = (alpha_post - 1) / (alpha_post + beta_post - 2)
mean_est = alpha_post / (alpha_post + beta_post)

print(f"MLE:         {mle:.3f}")   # 0.700
print(f"MAP (mode):  {map_est:.3f}")   # 0.667
print(f"Posterior mean: {mean_est:.3f}")   # 0.643

Bayesian vs frequentist

Frequentist (MLE)Bayesian
θ\theta is...Fixed unknown constantRandom variable
OutputPoint estimateFull posterior distribution
UncertaintyConfidence intervals (tricky to interpret)Credible intervals (direct probability)
Prior knowledgeCannot incorporateNaturally incorporated
ScalabilityFastCan be slow (posterior integration)

Predictive distribution

Rather than a single θ^\hat{\theta}, Bayesian models make predictions by averaging over all parameters weighted by the posterior:

p(xnew∣D)=∫p(xnew∣θ) p(θ∣D) dθp(x_\text{new} | \mathcal{D}) = \int p(x_\text{new} | \theta)\, p(\theta | \mathcal{D})\, d\theta

This marginalizes out θ\theta and naturally quantifies uncertainty in the prediction. In practice, this integral is often approximated with variational inference or MCMC.

When the posterior has no conjugate form, MCMC samples it directly — knowing the density only up to its normalizing constant:

Deep dive
MCMC Sampling: Metropolis–Hastings & Gibbs
Includes runnable notebook

Bayesian deep learning

Full Bayesian inference over neural network weights is computationally intractable. Practical approximations include: dropout as approximate Bayes (Gal & Ghahramani, 2016), variational inference (Bayes by Backprop), and deep ensembles (train multiple networks from different initializations).

Exercise

P(Disease | Positive test) is the posterior. Which theorem connects it to P(Positive | Disease)?

Need a hint?

We need to 'flip' the conditional.

Exercise

MAP estimation with a Gaussian prior N(0, τ²) on weights is equivalent to what regularization?

Need a hint?

The log of a Gaussian prior involves the squared norm of the weights.

Exercise

You use a Beta(2, 2) prior for a coin's bias p. After observing 6 heads and 4 tails, what is the posterior?

Need a hint?

For Beta-Bernoulli conjugacy: posterior = Beta(α + heads, β + tails).

Common mistakes

  1. Choosing an uninformative prior carelessly — a flat prior over a parameter is not uninformative after a change of variables (Jeffreys prior addresses this).
  2. Treating the posterior mean as the answer without acknowledging posterior width — narrow posteriors can be overconfident on small datasets.
  3. Confusing MAP estimation (maximizes the posterior, ignores uncertainty) with full Bayesian inference (integrates over the posterior).

Key takeaways

  • Bayesian inference treats model parameters as random variables with a prior distribution.
  • The posterior = likelihood × prior (normalized) — it combines data evidence with prior beliefs.
  • MAP = MLE + log-prior; Gaussian prior → L2 regularization; Laplace prior → L1 regularization.
  • The predictive distribution averages over parameters, naturally capturing uncertainty.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Bayesian InferenceThinking in ProbabilitiesProbability DistributionsMaximum Likelihood Estima…Gaussian Mixture ModelsHypothesis Testing & p-va…Regularization: Ridge & L…Beta-Bernoulli Conjugate …MCMC Sampling: Metropolis…Hypothesis Testing: Null,…Information Theory for ML…MLE for the Gaussian