Maximum Likelihood Estimation
You flip a bent coin 10 times and get 7 heads. What is the coin's true probability of heads? Your gut says , but why is that the best answer, and not or ?
Maximum Likelihood Estimation (MLE) turns that gut feeling into a principle: out of every possible coin (every value of ), pick the one that makes the data you actually saw as probable as possible. The same idea fits a Gaussian to measurements, fits a logistic regression to labels, and underlies almost every loss function in machine learning. This is where the loop's objective stage comes from: a loss is usually just a negative log-likelihood in disguise.
Real-world examples: Fitting a linear regression model, training a logistic classifier, and learning the parameters of a Gaussian mixture — all minimize the negative log-likelihood, i.e. maximize the likelihood of the data.
The likelihood function
Suppose your data are drawn independently from a distribution , where collects the unknown parameters (the coin's , or a Gaussian's and ).
Because the draws are independent, the probability of seeing all of them is the product of the individual probabilities. Viewed as a function of with the data held fixed, that product is the likelihood:
MLE picks the that maximizes it:
Likelihood is not a probability over θ
varies with the data fixed. It is not a probability distribution over . It does not integrate to 1 over . To get probabilities over you need a prior, which is Bayesian inference (next lesson).
Why we maximize the log-likelihood
Multiplying numbers between 0 and 1 produces a vanishingly small product (think ), which underflows to exactly on a computer. Taking the logarithm fixes this and, because turns products into sums, makes the calculus far easier:
The log is strictly increasing, so it never moves the location of the maximum:
The recipe for the rest of this lesson is always the same: write , differentiate, set the derivative to zero, and solve.
Maximum likelihood for a Gaussian
Log-likelihood = how probable the observed sample is under the candidate Gaussian. It is maximized exactly when μ = sample mean and σ = sample std — that's the MLE.
Worked derivation 1: Gaussian mean and variance
The MLE for a Gaussian follows an 8-step recipe: write the log-likelihood, differentiate with respect to (steps 3–5) and with respect to (steps 6–8), set each derivative to zero, and solve. The results are (sample mean) and (biased variance with divisor , not ). For the full algebraic derivation, a numeric verification on , and an explanation of the bias, see:
Worked derivation 2: Bernoulli p
Model: each with . A compact way to write the probability of one outcome is
Check it: if this is ; if it is .
Step 1: log a single term (the exponents drop down as multipliers):
Step 2: sum over all points. Let be the number of successes:
Step 3: differentiate, using :
Step 4: set to zero and cross-multiply by :
Step 5: expand and solve. The terms cancel:
So the MLE is the empirical fraction of successes. For the bent coin (7 heads in 10 tosses) this is exactly , confirming the gut feeling from the start of the lesson.
MLE = minimizing negative log-likelihood = cross-entropy
Optimizers minimize, so in practice we minimize the negative log-likelihood (NLL) instead of maximizing — same answer, flipped sign:
Now take the Bernoulli NLL and divide by to make it a per-example average. Writing the model's predicted probability as :
That right-hand side is literally the definition of binary cross-entropy loss . There is nothing to prove beyond recognizing the two formulas are identical:
Therefore minimizing cross-entropy is exactly maximizing the Bernoulli log-likelihood of the labels. Every time you train a classifier with cross-entropy, you are doing MLE. For more than two classes, the same argument with a categorical distribution gives categorical cross-entropy.
Loss functions are MLE objectives in disguise
Pick a noise/label distribution, write its negative log-likelihood, and you recover a familiar loss:
- MSE = MLE for Gaussian noise (derivation 1 above)
- Binary cross-entropy = MLE for Bernoulli (derivation 2 above)
- Categorical cross-entropy = MLE for a Categorical distribution
- MAE = MLE for Laplace noise
import numpy as np
# MLE for a Bernoulli — bent coin with unknown p
data = np.array([1, 1, 0, 1, 1, 0, 1, 0, 1, 1]) # 7 heads, 3 tails
def log_likelihood(p, data):
eps = 1e-12 # avoid log(0)
return np.sum(data * np.log(p + eps) + (1 - data) * np.log(1 - p + eps))
# Brute force: scan a grid of candidate p and keep the argmax
grid = np.linspace(0.01, 0.99, 9999)
ll = np.array([log_likelihood(p, data) for p in grid])
p_grid = grid[ll.argmax()]
print("Grid-search MLE: p = {:.3f}".format(p_grid)) # ~0.700
print("Closed-form k/n: p = {:.3f}".format(data.mean())) # 0.700 — same!
Properties of MLE
| Property | Description |
|---|---|
| Consistent | as |
| Asymptotically efficient | Achieves the Cramér–Rao lower bound for large |
| Invariant | for any invertible transform |
| Biased | May be biased in finite samples (e.g., Gaussian variance) |
To reason about uncertainty in itself (to put a probability distribution on the parameters) you need a prior, which is Bayesian inference (next lesson).
The information-theoretic view of the same objects — entropy, KL divergence, and why minimizing cross-entropy minimizes the KL to the data distribution — is developed in the deep dive:
Exercise
MLE for the mean of a Gaussian distribution gives which estimate?
Need a hint?
Maximize the log-likelihood by setting the derivative to zero.
Exercise
Minimizing mean squared error (MSE) is equivalent to MLE under what assumption?
Need a hint?
Which distribution has MSE as its negative log-likelihood?
Exercise
You flip a coin 10 times and get 7 heads. What is the MLE estimate of p (probability of heads)?
Need a hint?
MLE for Bernoulli is the empirical frequency.
Common mistakes
- Confusing likelihood with probability — is a function of for fixed , not a probability over .
- Maximizing likelihood on training data without regularization leads to overfitting — MLE can perfectly interpolate noise.
- Forgetting to check that the MLE objective is concave (or using gradient descent for a non-concave likelihood) — multiple local optima can exist.
Key takeaways
- MLE chooses parameters that maximize the probability of the observed data: .
- Log-likelihood turns the product into a sum, avoids underflow, and is what you actually differentiate.
- Gaussian MLE → and (biased, divisor ).
- Bernoulli MLE → , the empirical success rate.
- Maximizing log-likelihood = minimizing NLL; for a Bernoulli that NLL is cross-entropy, so MSE and cross-entropy are MLE objectives in disguise.
- MLE is consistent and asymptotically efficient, but says nothing about parameter uncertainty.
Related concepts
- Probability Distributions: the distributions MLE fits
- Bayesian Inference: adds a prior to MLE
- Gradient Descent: the optimizer that maximizes likelihood in deep models