MLE for the Gaussian
Given i.i.d. observations from a Gaussian , we want the parameter values that maximize the probability of seeing this exact data. The derivation has eight steps: write the log-likelihood, differentiate, and solve separately for and .
Setting up the Gaussian density and log-likelihood
The density of a single observation is:
Step 1: log a single term. The cancels the :
Step 2: sum over all points to get the full log-likelihood:
Solving for the maximum-likelihood mean
Step 3: differentiate with respect to .
The first term has no . The chain rule on gives :
Step 4: set the derivative to zero.
Since the prefactor cannot make it zero, so the sum must vanish:
Step 5: solve for :
The MLE for the mean is the sample average.
Gaussian MLE = OLS
Maximizing over with fixed is the same as minimizing . So fitting a linear model by minimizing squared error is exactly MLE under Gaussian noise. The two derivations (OLS and Gaussian MLE) are the same optimization in disguise.
Solving for the maximum-likelihood variance
It is cleanest to treat as the variable.
Step 6: rewrite and differentiate with respect to .
Using and :
Step 7 — set to zero and multiply through by :
Step 8: solve for , substituting :
Verifying the estimates numerically
Take (eight observations from Rice & Nerman's classic example):
So and (standard deviation ).
import numpy as np
x = np.array([2, 4, 4, 4, 5, 5, 7, 9], dtype=float)
mu_mle = x.mean()
sigma2_mle = ((x - mu_mle) ** 2).mean() # divisor n (MLE, biased)
sigma2_unbiased = ((x - mu_mle) ** 2).sum() / (len(x) - 1) # divisor n-1
print(f"MLE mean: {mu_mle:.3f}") # 5.000
print(f"MLE variance (n): {sigma2_mle:.3f}") # 4.000
print(f"Unbiased variance (n-1): {sigma2_unbiased:.3f}") # 4.571
print(f"Numpy default std²: {x.var(ddof=1):.3f}") # 4.571 (ddof=1 = n-1)
Why the MLE variance is biased
The MLE variance divides by , not . Why is it biased?
Because we substituted (an estimate fit to the same data), the squared deviations come out systematically too small. The true deviations would be larger since the true doesn't chase the sample.
Formally, .
The unbiased correction uses (Bessel's correction):
MLE is consistent (as , ) but it is not unbiased in finite samples.
The log-likelihood surface
The log-likelihood is a concave function of , so the unique maximum is the MLE:
import numpy as np
import matplotlib.pyplot as plt
x = np.array([2, 4, 4, 4, 5, 5, 7, 9], dtype=float)
mu_grid = np.linspace(3, 7, 100)
v_grid = np.linspace(1, 9, 100)
MU, V = np.meshgrid(mu_grid, v_grid)
# Log-likelihood on grid
LL = np.zeros_like(MU)
for xi in x:
LL += -0.5 * np.log(2 * np.pi * V) - (xi - MU) ** 2 / (2 * V)
plt.figure(figsize=(7, 5))
plt.contourf(MU, V, LL, levels=30, cmap='viridis')
plt.colorbar(label='log-likelihood')
plt.scatter([x.mean()], [x.var()], color='red', s=100, zorder=5, label='MLE')
plt.xlabel(r'$\mu$')
plt.ylabel(r'$\sigma^2$')
plt.title('Gaussian log-likelihood surface')
plt.legend()
plt.tight_layout()
plt.show()
Key results at a glance
| Parameter | MLE formula | Notes |
|---|---|---|
| Mean | Unbiased | |
| Variance | Biased by factor |
Related concepts
- MLE & the Bernoulli: companion derivation for discrete data; cross-entropy connection
- Bayesian Inference: adds a prior; posterior mean shifts toward the prior
- OLS: The Normal Equation: minimizing squared error = MLE under Gaussian noise