Beta-Bernoulli Conjugate PosteriorOpen in Colab

Beta-Bernoulli Conjugate Posterior

The Beta-Bernoulli model is the cleanest example of conjugacy: multiplying a Bernoulli likelihood by a Beta prior produces a Beta posterior. No integration is required — the update reduces to adding counts to the prior's parameters.

Model, prior, and setup

Model: each flip xi∈{0,1}x_i \in \{0,1\} is drawn from Bernoulli(p)\text{Bernoulli}(p), where the bias p∈[0,1]p \in [0,1] is unknown.

Prior: p∼Beta(α,β)p \sim \text{Beta}(\alpha, \beta), whose density is:

p(p)=1B(α,β) pα−1(1−p)β−1p(p) = \frac{1}{B(\alpha,\beta)}\,p^{\alpha-1}(1-p)^{\beta-1}

where B(α,β)=Γ(α)Γ(β)/Γ(α+β)B(\alpha,\beta) = \Gamma(\alpha)\Gamma(\beta)/\Gamma(\alpha+\beta) is the Beta function — a constant that does not depend on pp.

Deriving the posterior in five steps

Step 1: write the likelihood. For nn independent flips with kk heads and n−kn-k tails:

p(D∣p)=∏i=1npxi(1−p)1−xi=pk(1−p)n−kp(\mathcal{D}\mid p) = \prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} = p^k(1-p)^{n-k}

Step 2: write the prior.

p(p)  ∝  pα−1(1−p)β−1p(p) \;\propto\; p^{\alpha-1}(1-p)^{\beta-1}

(Dropping the normalizing constant 1/B(α,β)1/B(\alpha,\beta) since it does not contain pp.)

Step 3: multiply (unnormalized Bayes).

p(p∣D)  ∝  pk(1−p)n−k⏟likelihood⋅pα−1(1−p)β−1⏟priorp(p\mid\mathcal{D}) \;\propto\; \underbrace{p^k(1-p)^{n-k}}_{\text{likelihood}} \cdot \underbrace{p^{\alpha-1}(1-p)^{\beta-1}}_{\text{prior}}

Step 4: collect exponents. Add the powers of pp and of (1−p)(1-p) separately:

p(p∣D)  ∝  p(α+k)−1 (1−p)(β+n−k)−1p(p\mid\mathcal{D}) \;\propto\; p^{(\alpha+k)-1}\,(1-p)^{(\beta+n-k)-1}

Step 5: recognize the shape. This is exactly the kernel of a Beta(α+k,  β+n−k)\text{Beta}(\alpha+k,\;\beta+n-k) density. Since a probability distribution is uniquely determined by its kernel, the normalizing constant must be B(α+k,  β+n−k)B(\alpha+k,\;\beta+n-k), and therefore:

  p∣D  ∼  Beta(α+k,  β+n−k)  \boxed{\;p\mid\mathcal{D}\;\sim\;\text{Beta}(\alpha+k,\;\beta+n-k)\;}

No integration was required — conjugacy did the work. Writing n1=kn_1 = k (heads) and n0=n−kn_0 = n-k (tails) gives the memorable update rule:

Beta(α, β)  →  +n1 heads,  +n0 tails  Beta(α+n1,  β+n0)\text{Beta}(\alpha,\,\beta) \;\xrightarrow{\;+n_1\text{ heads},\;+n_0\text{ tails}}\; \text{Beta}(\alpha+n_1,\;\beta+n_0)

α and β as pseudo-counts

The prior parameters α\alpha and β\beta act like imaginary data: α−1\alpha-1 prior heads and β−1\beta-1 prior tails (using the mode formula). A Beta(1,1)\text{Beta}(1,1) prior (uniform) starts with zero pseudo-evidence and lets the data speak entirely. A Beta(10,10)\text{Beta}(10,10) prior encodes strong belief that p≈0.5p\approx0.5 and resists updating until many flips arrive.

A numeric update instance

Prior: Beta(2,2)\text{Beta}(2,2) — a mild "probably fair coin" belief, peaked at p=0.5p=0.5.

Observation: k=7k=7 heads, n−k=3n-k=3 tails in n=10n=10 flips.

Posterior update:

αpost=2+7=9,βpost=2+3=5\alpha_\text{post} = 2 + 7 = 9, \qquad \beta_\text{post} = 2 + 3 = 5 p∣D  ∼  Beta(9,5)p\mid\mathcal{D} \;\sim\; \text{Beta}(9,5)

Three summaries of the posterior:

SummaryFormulaValue
Mode (MAP estimate)(α−1)/(α+β−2)(\alpha-1)/(\alpha+\beta-2)(9−1)/(9+5−2)=8/12≈0.667(9-1)/(9+5-2) = 8/12 \approx 0.667
Mean (posterior mean)α/(α+β)\alpha/(\alpha+\beta)9/14≈0.6439/14 \approx 0.643
MLE (no prior)k/nk/n7/10=0.7007/10 = 0.700

The prior pulls the estimate away from the MLE toward 0.50.5. As n→∞n\to\infty the +2+2 corrections become negligible and all three converge to the MLE.

Python
import numpy as np
from scipy.stats import beta as beta_dist
import matplotlib.pyplot as plt

alpha_prior, beta_prior = 2, 2
k, n = 7, 10
n0 = n - k

alpha_post = alpha_prior + k
beta_post  = beta_prior + n0

print(f"Prior:     Beta({alpha_prior}, {beta_prior})")
print(f"Posterior: Beta({alpha_post}, {beta_post})")
print(f"MAP:       {(alpha_post-1)/(alpha_post+beta_post-2):.4f}")
print(f"Mean:      {alpha_post/(alpha_post+beta_post):.4f}")
print(f"MLE:       {k/n:.4f}")

Sequential updating

Because the posterior is a Beta, it becomes the prior for the next observation. After seeing flips one by one, the posterior updates incrementally:

Python
import numpy as np
from scipy.stats import beta as beta_dist
import matplotlib.pyplot as plt

plt.rcParams.update({'figure.facecolor': '#0f1117', 'axes.facecolor': '#1a1d27',
                     'text.color': '#eee', 'axes.labelcolor': '#ccc'})

flips = [1,1,0,1,1,0,1,0,1,1]  # 7 heads, 3 tails
alpha, beta_ = 2, 2              # initial prior

p = np.linspace(0, 1, 300)
fig, axes = plt.subplots(2, 5, figsize=(14, 5))
axes = axes.ravel()

for i, x in enumerate(flips):
    alpha += x
    beta_ += (1 - x)
    axes[i].fill_between(p, beta_dist.pdf(p, alpha, beta_),
                         color='#6366f1', alpha=0.7)
    axes[i].axvline(alpha/(alpha+beta_), color='#ef4444', linestyle='--', lw=1.5)
    axes[i].set_title(f"After flip {i+1}: Beta({alpha},{beta_})", fontsize=8)
    axes[i].set_yticks([])

plt.suptitle("Sequential Beta posterior updating", color='#eee')
plt.tight_layout()
plt.show()

Modeling different prior beliefs with the Beta

Prior Beta(α,β)\text{Beta}(\alpha,\beta)Meaning
Beta(1,1)\text{Beta}(1,1)Uniform: complete ignorance
Beta(2,2)\text{Beta}(2,2)Weak belief in fairness
Beta(10,10)\text{Beta}(10,10)Strong belief in fairness
Beta(0.5,0.5)\text{Beta}(0.5,0.5)U-shaped: suspect extreme bias
Beta(k+1,n−k+1)\text{Beta}(k+1, n-k+1)Equivalent to nn observations with Laplace smoothing

Connection to MAP estimation

MAP estimation finds the mode of the posterior. For Beta(α+k, β+n−k)\text{Beta}(\alpha+k,\,\beta+n-k) with α,β>1\alpha,\beta > 1:

p^MAP=(α+k)−1(α+k)+(β+n−k)−2=α+k−1α+β+n−2\hat{p}_\text{MAP} = \frac{(\alpha+k) - 1}{(\alpha+k)+(\beta+n-k) - 2} = \frac{\alpha+k-1}{\alpha+\beta+n-2}

With the uniform prior Beta(1,1)\text{Beta}(1,1): p^MAP=k/n=p^MLE\hat{p}_\text{MAP} = k/n = \hat{p}_\text{MLE}. With an informative prior, MAP pulls the estimate toward the prior mode (α−1)/(α+β−2)(\alpha-1)/(\alpha+\beta-2).

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Beta-Bernoulli Conjugate PosteriorMLE for the GaussianMCMC Sampling: Metropolis…Probability Distributions…Ridge & Lasso Regularizat…The Gaussian Integral (No…Bayesian InferenceProbability Distributions