Beta-Bernoulli Conjugate Posterior
The Beta-Bernoulli model is the cleanest example of conjugacy: multiplying a Bernoulli likelihood by a Beta prior produces a Beta posterior. No integration is required — the update reduces to adding counts to the prior's parameters.
Model, prior, and setup
Model: each flip is drawn from , where the bias is unknown.
Prior: , whose density is:
where is the Beta function — a constant that does not depend on .
Deriving the posterior in five steps
Step 1: write the likelihood. For independent flips with heads and tails:
Step 2: write the prior.
(Dropping the normalizing constant since it does not contain .)
Step 3: multiply (unnormalized Bayes).
Step 4: collect exponents. Add the powers of and of separately:
Step 5: recognize the shape. This is exactly the kernel of a density. Since a probability distribution is uniquely determined by its kernel, the normalizing constant must be , and therefore:
No integration was required — conjugacy did the work. Writing (heads) and (tails) gives the memorable update rule:
α and β as pseudo-counts
The prior parameters and act like imaginary data: prior heads and prior tails (using the mode formula). A prior (uniform) starts with zero pseudo-evidence and lets the data speak entirely. A prior encodes strong belief that and resists updating until many flips arrive.
A numeric update instance
Prior: — a mild "probably fair coin" belief, peaked at .
Observation: heads, tails in flips.
Posterior update:
Three summaries of the posterior:
| Summary | Formula | Value |
|---|---|---|
| Mode (MAP estimate) | ||
| Mean (posterior mean) | ||
| MLE (no prior) |
The prior pulls the estimate away from the MLE toward . As the corrections become negligible and all three converge to the MLE.
import numpy as np
from scipy.stats import beta as beta_dist
import matplotlib.pyplot as plt
alpha_prior, beta_prior = 2, 2
k, n = 7, 10
n0 = n - k
alpha_post = alpha_prior + k
beta_post = beta_prior + n0
print(f"Prior: Beta({alpha_prior}, {beta_prior})")
print(f"Posterior: Beta({alpha_post}, {beta_post})")
print(f"MAP: {(alpha_post-1)/(alpha_post+beta_post-2):.4f}")
print(f"Mean: {alpha_post/(alpha_post+beta_post):.4f}")
print(f"MLE: {k/n:.4f}")
Sequential updating
Because the posterior is a Beta, it becomes the prior for the next observation. After seeing flips one by one, the posterior updates incrementally:
import numpy as np
from scipy.stats import beta as beta_dist
import matplotlib.pyplot as plt
plt.rcParams.update({'figure.facecolor': '#0f1117', 'axes.facecolor': '#1a1d27',
'text.color': '#eee', 'axes.labelcolor': '#ccc'})
flips = [1,1,0,1,1,0,1,0,1,1] # 7 heads, 3 tails
alpha, beta_ = 2, 2 # initial prior
p = np.linspace(0, 1, 300)
fig, axes = plt.subplots(2, 5, figsize=(14, 5))
axes = axes.ravel()
for i, x in enumerate(flips):
alpha += x
beta_ += (1 - x)
axes[i].fill_between(p, beta_dist.pdf(p, alpha, beta_),
color='#6366f1', alpha=0.7)
axes[i].axvline(alpha/(alpha+beta_), color='#ef4444', linestyle='--', lw=1.5)
axes[i].set_title(f"After flip {i+1}: Beta({alpha},{beta_})", fontsize=8)
axes[i].set_yticks([])
plt.suptitle("Sequential Beta posterior updating", color='#eee')
plt.tight_layout()
plt.show()
Modeling different prior beliefs with the Beta
| Prior | Meaning |
|---|---|
| Uniform: complete ignorance | |
| Weak belief in fairness | |
| Strong belief in fairness | |
| U-shaped: suspect extreme bias | |
| Equivalent to observations with Laplace smoothing |
Connection to MAP estimation
MAP estimation finds the mode of the posterior. For with :
With the uniform prior : . With an informative prior, MAP pulls the estimate toward the prior mode .
Related concepts
- Bayesian Inference: the full lesson: Bayes' theorem, MAP, L2-regularization connection
- MLE for the Gaussian: the frequentist alternative (no prior)
- Probability Distributions: Beta, Bernoulli, and the exponential family