Maximum Likelihood EstimationOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Maximum Likelihood Estimation

You flip a bent coin 10 times and get 7 heads. What is the coin's true probability of heads? Your gut says 0.70.7, but why is that the best answer, and not 0.60.6 or 0.80.8?

Maximum Likelihood Estimation (MLE) turns that gut feeling into a principle: out of every possible coin (every value of pp), pick the one that makes the data you actually saw as probable as possible. The same idea fits a Gaussian to measurements, fits a logistic regression to labels, and underlies almost every loss function in machine learning. This is where the loop's objective stage comes from: a loss is usually just a negative log-likelihood in disguise.

Real-world examples: Fitting a linear regression model, training a logistic classifier, and learning the parameters of a Gaussian mixture — all minimize the negative log-likelihood, i.e. maximize the likelihood of the data.

The likelihood function

Suppose your data D={x1,…,xn}\mathcal{D} = \{x_1, \ldots, x_n\} are drawn independently from a distribution p(x∣θ)p(x \mid \theta), where θ\theta collects the unknown parameters (the coin's pp, or a Gaussian's μ\mu and σ2\sigma^2).

Because the draws are independent, the probability of seeing all of them is the product of the individual probabilities. Viewed as a function of θ\theta with the data held fixed, that product is the likelihood:

L(θ)=p(D∣θ)=∏i=1np(xi∣θ)L(\theta) = p(\mathcal{D} \mid \theta) = \prod_{i=1}^n p(x_i \mid \theta)

MLE picks the θ\theta that maximizes it:

θ^MLE=arg⁡max⁡θ  L(θ)\hat{\theta}_\text{MLE} = \arg\max_\theta \; L(\theta)

Likelihood is not a probability over θ

L(θ)L(\theta) varies θ\theta with the data fixed. It is not a probability distribution over θ\theta. It does not integrate to 1 over θ\theta. To get probabilities over θ\theta you need a prior, which is Bayesian inference (next lesson).

Why we maximize the log-likelihood

Multiplying nn numbers between 0 and 1 produces a vanishingly small product (think 0.110000.1^{1000}), which underflows to exactly 0.00.0 on a computer. Taking the logarithm fixes this and, because ln⁡\ln turns products into sums, makes the calculus far easier:

ℓ(θ)=ln⁡L(θ)=ln⁡∏i=1np(xi∣θ)=∑i=1nln⁡p(xi∣θ)\ell(\theta) = \ln L(\theta) = \ln \prod_{i=1}^n p(x_i \mid \theta) = \sum_{i=1}^n \ln p(x_i \mid \theta)

The log is strictly increasing, so it never moves the location of the maximum:

arg⁡max⁡θ  ℓ(θ)=arg⁡max⁡θ  L(θ)\arg\max_\theta \; \ell(\theta) = \arg\max_\theta \; L(\theta)

The recipe for the rest of this lesson is always the same: write ℓ(θ)\ell(\theta), differentiate, set the derivative to zero, and solve.

Maximum likelihood for a Gaussian

x (observed values)density
log-likelihood-38.04
MLE μ̂0.71
MLE σ̂1.21

Log-likelihood = how probable the observed sample is under the candidate Gaussian. It is maximized exactly when μ = sample mean and σ = sample std — that's the MLE.

Worked derivation 1: Gaussian mean and variance

The MLE for a Gaussian follows an 8-step recipe: write the log-likelihood, differentiate with respect to μ\mu (steps 3–5) and with respect to σ2=v\sigma^2 = v (steps 6–8), set each derivative to zero, and solve. The results are μ^=xˉ\hat{\mu} = \bar{x} (sample mean) and σ^2=1n∑(xi−xˉ)2\hat{\sigma}^2 = \frac{1}{n}\sum(x_i-\bar{x})^2 (biased variance with divisor nn, not n−1n-1). For the full algebraic derivation, a numeric verification on x=[2,4,4,4,5,5,7,9]\mathbf{x} = [2,4,4,4,5,5,7,9], and an explanation of the bias, see:

Deep dive
MLE for the Gaussian
Includes runnable notebook

Worked derivation 2: Bernoulli p

Model: each xi∈{0,1}x_i \in \{0, 1\} with xi∼Bernoulli(p)x_i \sim \text{Bernoulli}(p). A compact way to write the probability of one outcome is

p(xi∣p)=pxi(1−p)1−xip(x_i \mid p) = p^{x_i}(1-p)^{1-x_i}

Check it: if xi=1x_i = 1 this is pp; if xi=0x_i = 0 it is 1−p1-p.

Step 1: log a single term (the exponents drop down as multipliers):

ln⁡p(xi∣p)=xiln⁡p+(1−xi)ln⁡(1−p)\ln p(x_i \mid p) = x_i \ln p + (1 - x_i)\ln(1-p)

Step 2: sum over all points. Let k=∑ixik = \sum_i x_i be the number of successes:

ℓ(p)=∑i=1n[xiln⁡p+(1−xi)ln⁡(1−p)]=kln⁡p+(n−k)ln⁡(1−p)\ell(p) = \sum_{i=1}^n \big[x_i \ln p + (1-x_i)\ln(1-p)\big] = k\ln p + (n-k)\ln(1-p)

Step 3: differentiate, using ddpln⁡(1−p)=−11−p\frac{d}{dp}\ln(1-p) = -\frac{1}{1-p}:

dℓdp=kp−n−k1−p\frac{d\ell}{dp} = \frac{k}{p} - \frac{n-k}{1-p}

Step 4: set to zero and cross-multiply by p(1−p)p(1-p):

kp=n−k1−p    ⟹    k(1−p)=(n−k)p\frac{k}{p} = \frac{n-k}{1-p} \;\;\Longrightarrow\;\; k(1-p) = (n-k)p

Step 5: expand and solve. The kpkp terms cancel:

k−kp=np−kp    ⟹    k=np    ⟹      p^MLE=kn=1n∑i=1nxi  k - kp = np - kp \;\;\Longrightarrow\;\; k = np \;\;\Longrightarrow\;\; \boxed{\;\hat{p}_\text{MLE} = \frac{k}{n} = \frac{1}{n}\sum_{i=1}^n x_i\;}

So the MLE is the empirical fraction of successes. For the bent coin (7 heads in 10 tosses) this is exactly p^=0.7\hat p = 0.7, confirming the gut feeling from the start of the lesson.

MLE = minimizing negative log-likelihood = cross-entropy

Optimizers minimize, so in practice we minimize the negative log-likelihood (NLL) instead of maximizing ℓ\ell — same answer, flipped sign:

θ^MLE=arg⁡max⁡θℓ(θ)=arg⁡min⁡θ[−ℓ(θ)]=arg⁡min⁡θ  NLL(θ)\hat\theta_\text{MLE} = \arg\max_\theta \ell(\theta) = \arg\min_\theta \big[-\ell(\theta)\big] = \arg\min_\theta \;\text{NLL}(\theta)

Now take the Bernoulli NLL and divide by nn to make it a per-example average. Writing the model's predicted probability as y^i=p(yi=1∣xi,θ)\hat y_i = p(y_i = 1 \mid x_i, \theta):

1n NLL(θ)=−1n∑i=1n[yiln⁡y^i+(1−yi)ln⁡(1−y^i)]\frac{1}{n}\,\text{NLL}(\theta) = -\frac{1}{n}\sum_{i=1}^n \big[y_i \ln \hat{y}_i + (1-y_i)\ln(1-\hat{y}_i)\big]

That right-hand side is literally the definition of binary cross-entropy loss LCE\mathcal{L}_\text{CE}. There is nothing to prove beyond recognizing the two formulas are identical:

LCE=−1n ℓ(θ)\mathcal{L}_\text{CE} = -\frac{1}{n}\,\ell(\theta)

Therefore minimizing cross-entropy is exactly maximizing the Bernoulli log-likelihood of the labels. Every time you train a classifier with cross-entropy, you are doing MLE. For more than two classes, the same argument with a categorical distribution gives categorical cross-entropy.

Loss functions are MLE objectives in disguise

Pick a noise/label distribution, write its negative log-likelihood, and you recover a familiar loss:

  • MSE = MLE for Gaussian noise (derivation 1 above)
  • Binary cross-entropy = MLE for Bernoulli (derivation 2 above)
  • Categorical cross-entropy = MLE for a Categorical distribution
  • MAE = MLE for Laplace noise
Python
import numpy as np

# MLE for a Bernoulli — bent coin with unknown p
data = np.array([1, 1, 0, 1, 1, 0, 1, 0, 1, 1])  # 7 heads, 3 tails

def log_likelihood(p, data):
    eps = 1e-12   # avoid log(0)
    return np.sum(data * np.log(p + eps) + (1 - data) * np.log(1 - p + eps))

# Brute force: scan a grid of candidate p and keep the argmax
grid = np.linspace(0.01, 0.99, 9999)
ll = np.array([log_likelihood(p, data) for p in grid])
p_grid = grid[ll.argmax()]

print("Grid-search MLE:  p = {:.3f}".format(p_grid))   # ~0.700
print("Closed-form k/n:  p = {:.3f}".format(data.mean()))  # 0.700 — same!

Properties of MLE

PropertyDescription
Consistentθ^→θ∗\hat{\theta} \to \theta^* as n→∞n \to \infty
Asymptotically efficientAchieves the Cramér–Rao lower bound for large nn
Invariantf(θ)^=f(θ^)\widehat{f(\theta)} = f(\hat{\theta}) for any invertible transform
BiasedMay be biased in finite samples (e.g., Gaussian variance)

To reason about uncertainty in θ\theta itself (to put a probability distribution on the parameters) you need a prior, which is Bayesian inference (next lesson).

The information-theoretic view of the same objects — entropy, KL divergence, and why minimizing cross-entropy minimizes the KL to the data distribution — is developed in the deep dive:

Deep dive
Information Theory for ML: Entropy, KL Divergence, and Cross-Entropy
Includes runnable notebook

Exercise

MLE for the mean of a Gaussian distribution gives which estimate?

Need a hint?

Maximize the log-likelihood by setting the derivative to zero.

Exercise

Minimizing mean squared error (MSE) is equivalent to MLE under what assumption?

Need a hint?

Which distribution has MSE as its negative log-likelihood?

Exercise

You flip a coin 10 times and get 7 heads. What is the MLE estimate of p (probability of heads)?

Need a hint?

MLE for Bernoulli is the empirical frequency.

Common mistakes

  1. Confusing likelihood with probability — L(θ;x)=p(x∣θ)L(\theta; x) = p(x|\theta) is a function of θ\theta for fixed xx, not a probability over θ\theta.
  2. Maximizing likelihood on training data without regularization leads to overfitting — MLE can perfectly interpolate noise.
  3. Forgetting to check that the MLE objective is concave (or using gradient descent for a non-concave likelihood) — multiple local optima can exist.

Key takeaways

  • MLE chooses parameters that maximize the probability of the observed data: θ^=arg⁡max⁡θℓ(θ)\hat\theta = \arg\max_\theta \ell(\theta).
  • Log-likelihood turns the product into a sum, avoids underflow, and is what you actually differentiate.
  • Gaussian MLE → μ^=xˉ\hat\mu = \bar x and σ^2=1n∑(xi−xˉ)2\hat\sigma^2 = \frac1n\sum(x_i-\bar x)^2 (biased, divisor nn).
  • Bernoulli MLE → p^=k/n\hat p = k/n, the empirical success rate.
  • Maximizing log-likelihood = minimizing NLL; for a Bernoulli that NLL is cross-entropy, so MSE and cross-entropy are MLE objectives in disguise.
  • MLE is consistent and asymptotically efficient, but says nothing about parameter uncertainty.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Maximum Likelihood EstimationThinking in ProbabilitiesProbability DistributionsBayesian InferenceGradient DescentEntropy & KL DivergenceStatistical Inference & C…MLE for the GaussianInformation Theory for ML…