Entropy & KL Divergence
Every classifier you will ever train with cross-entropy loss is secretly doing information theory. This lesson builds the three quantities that run modern ML (entropy, cross-entropy, and KL divergence) from one primitive: surprise. All three show up in the loop's objective stage: cross-entropy is the classification loss, and KL divergence is the objective behind variational methods and distillation.
Surprise and entropy
If an event has probability , define its surprise as . Rare events are very surprising ( nats); certain events aren't surprising at all (). The log makes surprise additive: the surprise of two independent events is the sum of their surprises.
Entropy is the expected surprise of a distribution:
It measures how unpredictable a source is:
- A fair coin: nats (1 bit) — maximal for two outcomes
- A coin with : nats — almost no uncertainty
- A uniform die: — uniform distributions maximize entropy, certainty minimizes it
Entropy is also a compression limit: you cannot losslessly encode samples from in fewer than nats per symbol on average. You met entropy already as a split criterion in decision trees — information gain is just entropy reduction.
Cross-entropy: paying for the wrong codebook
Suppose data really comes from , but you model it with . Your average surprise is the cross-entropy:
This is exactly the classification loss: the data distribution puts probability 1 on the true label, so the sum collapses to — the log-loss of your model's predicted probability. Minimizing cross-entropy = maximizing log-likelihood. Same math, two vocabularies.
KL divergence: the gap between them
Cross-entropy splits into two parts:
KL divergence is the extra surprise you suffer for using when the truth is . Three properties to internalize:
- , with equality only when — it behaves like a distance from the truth
- It is not symmetric: , and the difference matters
- If where , the divergence is infinite — never assign zero probability to something that can happen (the same lesson as Laplace smoothing in Naive Bayes)
import numpy as np
p = np.array([0.7, 0.2, 0.1]) # truth
q = np.array([0.5, 0.3, 0.2]) # model
H_p = -(p * np.log(p)).sum() # entropy of the truth
H_pq = -(p * np.log(q)).sum() # cross-entropy of model on truth
kl = (p * np.log(p / q)).sum() # KL divergence
print(f"H(p) = {H_p:.4f}, H(p,q) = {H_pq:.4f}, KL = {kl:.4f}")
print(f"identity check: H(p) + KL = {H_p + kl:.4f}") # equals H(p,q)
Which direction? Mode-seeking vs mean-covering
Minimizing (forward, used by MLE) forces to cover everywhere has mass — spread out, or pay infinite penalty. Minimizing (reverse, used by variational inference and VAEs) lets lock onto one mode of and ignore the rest. Same formula, opposite personalities.
Steer the single-Gaussian model below, then let each button find its own best fit. Minimizing forward KL spreads wide to cover both peaks of the truth; minimizing reverse KL collapses onto the heavier peak and abandons the other.
Forward vs reverse KL: mode-covering vs mode-seeking
The truth p (grey) is bimodal; your model q (indigo) is a single Gaussian you steer with μ and σ. Forward KL(p‖q) blows up wherever p has mass but q doesn't, so its best fit spreads q to COVER both modes. Reverse KL(q‖p) blows up wherever q has mass but p doesn't, so its best fit LOCKS onto one mode and ignores the other. Same formula, opposite personalities.
Where you'll meet these again
- Cross-entropy loss: every softmax classifier and language model (perplexity is just exponentiated cross-entropy, as in t-SNE's perplexity)
- KL terms in objectives: the VAE's regularizer, policy-constraint terms in RLHF (keeping the fine-tuned model close to the base model)
- Information gain: entropy reduction as a tree-split criterion
Exercise
Your model q assigns probability 0 to a word that actually occurs in the data distribution p. What happens to the forward KL divergence D(p‖q)?
Need a hint?
Look at the term p(x) · log(p(x)/q(x)) when q(x) = 0 but p(x) > 0.
Exercise
The cross-entropy H(p, q) always decomposes as which of the following?
Need a hint?
What's the 'extra surprise' term for using the wrong codebook?
Key takeaways
- Entropy = expected surprise : uniform maximizes it, certainty zeroes it
- Cross-entropy = entropy + KL: training on log-loss minimizes the KL between data and model
- KL is non-negative, zero only at equality, asymmetric, and infinite when the model says "impossible" about something real
- Forward KL covers all modes; reverse KL picks one — the choice shapes generative models
Related concepts
- Maximum Likelihood Estimation: minimizing cross-entropy is MLE in disguise
- Decision Trees: information gain is entropy reduction
- Variational Autoencoders: reverse-KL at work in the ELBO