MLE for the GaussianOpen in Colab

MLE for the Gaussian

Given nn i.i.d. observations x1,…,xnx_1, \ldots, x_n from a Gaussian N(μ,σ2)\mathcal{N}(\mu, \sigma^2), we want the parameter values that maximize the probability of seeing this exact data. The derivation has eight steps: write the log-likelihood, differentiate, and solve separately for μ\mu and σ2\sigma^2.

Setting up the Gaussian density and log-likelihood

The density of a single observation is:

p(xi∣μ,σ2)=12πσ2exp⁡ ⁣(−(xi−μ)22σ2)p(x_i \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(x_i-\mu)^2}{2\sigma^2}\right)

Step 1: log a single term. The ln⁡\ln cancels the exp⁡\exp:

ln⁡p(xi∣μ,σ2)=−12ln⁡(2πσ2)−(xi−μ)22σ2\ln p(x_i \mid \mu, \sigma^2) = -\tfrac{1}{2}\ln(2\pi\sigma^2) - \frac{(x_i - \mu)^2}{2\sigma^2}

Step 2: sum over all nn points to get the full log-likelihood:

ℓ(μ,σ2)=−n2ln⁡(2πσ2)−12σ2∑i=1n(xi−μ)2\ell(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2

Solving for the maximum-likelihood mean

Step 3: differentiate with respect to μ\mu.

The first term has no μ\mu. The chain rule on (xi−μ)2(x_i - \mu)^2 gives −2(xi−μ)-2(x_i - \mu):

∂ℓ∂μ=−12σ2∑i=1n−2(xi−μ)=1σ2∑i=1n(xi−μ)\frac{\partial \ell}{\partial \mu} = -\frac{1}{2\sigma^2}\sum_{i=1}^n -2(x_i - \mu) = \frac{1}{\sigma^2}\sum_{i=1}^n (x_i - \mu)

Step 4: set the derivative to zero.

Since σ2>0\sigma^2 > 0 the prefactor cannot make it zero, so the sum must vanish:

∑i=1n(xi−μ)=0  ⟹  ∑i=1nxi−nμ=0\sum_{i=1}^n (x_i - \mu) = 0 \;\Longrightarrow\; \sum_{i=1}^n x_i - n\mu = 0

Step 5: solve for μ\mu:

  μ^MLE=1n∑i=1nxi=xˉ  \boxed{\;\hat{\mu}_\text{MLE} = \frac{1}{n}\sum_{i=1}^n x_i = \bar{x}\;}

The MLE for the mean is the sample average.

Gaussian MLE = OLS

Maximizing ℓ\ell over μ\mu with σ2\sigma^2 fixed is the same as minimizing ∑(xi−μ)2\sum(x_i - \mu)^2. So fitting a linear model by minimizing squared error is exactly MLE under Gaussian noise. The two derivations (OLS and Gaussian MLE) are the same optimization in disguise.

Solving for the maximum-likelihood variance

It is cleanest to treat v=σ2v = \sigma^2 as the variable.

Step 6: rewrite and differentiate with respect to vv.

ℓ(μ,v)=−n2ln⁡(2π)−n2ln⁡v−12v∑i=1n(xi−μ)2\ell(\mu, v) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln v - \frac{1}{2v}\sum_{i=1}^n (x_i - \mu)^2

Using ddvln⁡v=1v\frac{d}{dv}\ln v = \frac{1}{v} and ddv1v=−1v2\frac{d}{dv}\frac{1}{v} = -\frac{1}{v^2}:

∂ℓ∂v=−n2v+12v2∑i=1n(xi−μ)2\frac{\partial \ell}{\partial v} = -\frac{n}{2v} + \frac{1}{2v^2}\sum_{i=1}^n (x_i - \mu)^2

Step 7 — set to zero and multiply through by 2v22v^2:

−nv+∑i=1n(xi−μ)2=0-nv + \sum_{i=1}^n (x_i - \mu)^2 = 0

Step 8: solve for vv, substituting μ=μ^=xˉ\mu = \hat{\mu} = \bar{x}:

  σ^MLE2=1n∑i=1n(xi−xˉ)2  \boxed{\;\hat{\sigma}^2_\text{MLE} = \frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2\;}

Verifying the estimates numerically

Take x=[2,4,4,4,5,5,7,9]\mathbf{x} = [2, 4, 4, 4, 5, 5, 7, 9] (eight observations from Rice & Nerman's classic example):

xˉ=2+4+4+4+5+5+7+98=408=5\bar{x} = \frac{2+4+4+4+5+5+7+9}{8} = \frac{40}{8} = 5 σ^MLE2=18[(2−5)2+(4−5)2+(4−5)2+(4−5)2+(5−5)2+(5−5)2+(7−5)2+(9−5)2]\hat{\sigma}^2_\text{MLE} = \frac{1}{8}\bigl[(2-5)^2+(4-5)^2+(4-5)^2+(4-5)^2+(5-5)^2+(5-5)^2+(7-5)^2+(9-5)^2\bigr] =18[9+1+1+1+0+0+4+16]=328=4= \frac{1}{8}[9 + 1 + 1 + 1 + 0 + 0 + 4 + 16] = \frac{32}{8} = 4

So μ^=5\hat{\mu} = 5 and σ^2=4\hat{\sigma}^2 = 4 (standard deviation σ^=2\hat{\sigma} = 2).

Python
import numpy as np

x = np.array([2, 4, 4, 4, 5, 5, 7, 9], dtype=float)

mu_mle = x.mean()
sigma2_mle = ((x - mu_mle) ** 2).mean()   # divisor n (MLE, biased)
sigma2_unbiased = ((x - mu_mle) ** 2).sum() / (len(x) - 1)  # divisor n-1

print(f"MLE mean:              {mu_mle:.3f}")        # 5.000
print(f"MLE variance (n):      {sigma2_mle:.3f}")    # 4.000
print(f"Unbiased variance (n-1): {sigma2_unbiased:.3f}")  # 4.571
print(f"Numpy default std²:    {x.var(ddof=1):.3f}") # 4.571  (ddof=1 = n-1)

Why the MLE variance is biased

The MLE variance divides by nn, not n−1n-1. Why is it biased?

Because we substituted μ^=xˉ\hat{\mu} = \bar{x} (an estimate fit to the same data), the squared deviations (xi−xˉ)2(x_i - \bar{x})^2 come out systematically too small. The true deviations (xi−μ)2(x_i - \mu)^2 would be larger since the true μ\mu doesn't chase the sample.

Formally, E[σ^MLE2]=n−1nσ2<σ2E[\hat{\sigma}^2_\text{MLE}] = \frac{n-1}{n}\sigma^2 < \sigma^2.

The unbiased correction uses n−1n-1 (Bessel's correction):

s2=1n−1∑i=1n(xi−xˉ)2s^2 = \frac{1}{n-1}\sum_{i=1}^n (x_i - \bar{x})^2

MLE is consistent (as n→∞n \to \infty, σ^MLE2→σ2\hat{\sigma}^2_\text{MLE} \to \sigma^2) but it is not unbiased in finite samples.

The log-likelihood surface

The log-likelihood is a concave function of (μ,v)(\mu, v), so the unique maximum is the MLE:

Python
import numpy as np
import matplotlib.pyplot as plt

x = np.array([2, 4, 4, 4, 5, 5, 7, 9], dtype=float)

mu_grid = np.linspace(3, 7, 100)
v_grid = np.linspace(1, 9, 100)
MU, V = np.meshgrid(mu_grid, v_grid)

# Log-likelihood on grid
LL = np.zeros_like(MU)
for xi in x:
    LL += -0.5 * np.log(2 * np.pi * V) - (xi - MU) ** 2 / (2 * V)

plt.figure(figsize=(7, 5))
plt.contourf(MU, V, LL, levels=30, cmap='viridis')
plt.colorbar(label='log-likelihood')
plt.scatter([x.mean()], [x.var()], color='red', s=100, zorder=5, label='MLE')
plt.xlabel(r'$\mu$')
plt.ylabel(r'$\sigma^2$')
plt.title('Gaussian log-likelihood surface')
plt.legend()
plt.tight_layout()
plt.show()

Key results at a glance

ParameterMLE formulaNotes
Mean μ\muμ^=xˉ=1n∑xi\hat{\mu} = \bar{x} = \frac{1}{n}\sum x_iUnbiased
Variance σ2\sigma^2σ^2=1n∑(xi−xˉ)2\hat{\sigma}^2 = \frac{1}{n}\sum(x_i-\bar{x})^2Biased by factor n−1n\frac{n-1}{n}

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
MLE for the GaussianBeta-Bernoulli Conjugate …Logistic Regression: Grad…OLS: The Normal EquationThe Gaussian Integral (No…Maximum Likelihood Estima…Bayesian Inference