Entropy & KL DivergenceOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Entropy & KL Divergence

Every classifier you will ever train with cross-entropy loss is secretly doing information theory. This lesson builds the three quantities that run modern ML (entropy, cross-entropy, and KL divergence) from one primitive: surprise. All three show up in the loop's objective stage: cross-entropy is the classification loss, and KL divergence is the objective behind variational methods and distillation.

Surprise and entropy

If an event has probability pp, define its surprise as −log⁡p-\log p. Rare events are very surprising (−log⁡0.01≈4.6-\log 0.01 \approx 4.6 nats); certain events aren't surprising at all (−log⁡1=0-\log 1 = 0). The log makes surprise additive: the surprise of two independent events is the sum of their surprises.

Entropy is the expected surprise of a distribution:

H(p)=−∑xp(x)log⁡p(x)H(p) = -\sum_x p(x) \log p(x)

It measures how unpredictable a source is:

  • A fair coin: H=log⁡2≈0.69H = \log 2 \approx 0.69 nats (1 bit) — maximal for two outcomes
  • A coin with p(heads)=0.99p(\text{heads}) = 0.99: H≈0.056H \approx 0.056 nats — almost no uncertainty
  • A uniform die: log⁡6\log 6 — uniform distributions maximize entropy, certainty minimizes it

Entropy is also a compression limit: you cannot losslessly encode samples from pp in fewer than H(p)H(p) nats per symbol on average. You met entropy already as a split criterion in decision trees — information gain is just entropy reduction.

Cross-entropy: paying for the wrong codebook

Suppose data really comes from pp, but you model it with qq. Your average surprise is the cross-entropy:

H(p,q)=−∑xp(x)log⁡q(x)H(p, q) = -\sum_x p(x) \log q(x)

This is exactly the classification loss: the data distribution puts probability 1 on the true label, so the sum collapses to −log⁡q(true label)-\log q(\text{true label}) — the log-loss of your model's predicted probability. Minimizing cross-entropy = maximizing log-likelihood. Same math, two vocabularies.

KL divergence: the gap between them

Cross-entropy splits into two parts:

H(p,q)=H(p)+DKL(p ∥ q)⏟your modeling penaltyH(p, q) = H(p) + \underbrace{D_{\mathrm{KL}}(p \,\|\, q)}_{\text{your modeling penalty}} DKL(p ∥ q)=∑xp(x)log⁡p(x)q(x)D_{\mathrm{KL}}(p \,\|\, q) = \sum_x p(x) \log \frac{p(x)}{q(x)}

KL divergence is the extra surprise you suffer for using qq when the truth is pp. Three properties to internalize:

  1. DKL≥0D_{\mathrm{KL}} \ge 0, with equality only when p=qp = q — it behaves like a distance from the truth
  2. It is not symmetric: DKL(p∥q)≠DKL(q∥p)D_{\mathrm{KL}}(p\|q) \ne D_{\mathrm{KL}}(q\|p), and the difference matters
  3. If q(x)=0q(x) = 0 where p(x)>0p(x) > 0, the divergence is infinite — never assign zero probability to something that can happen (the same lesson as Laplace smoothing in Naive Bayes)
Python
import numpy as np

p = np.array([0.7, 0.2, 0.1])          # truth
q = np.array([0.5, 0.3, 0.2])          # model

H_p  = -(p * np.log(p)).sum()          # entropy of the truth
H_pq = -(p * np.log(q)).sum()          # cross-entropy of model on truth
kl   = (p * np.log(p / q)).sum()       # KL divergence

print(f"H(p) = {H_p:.4f}, H(p,q) = {H_pq:.4f}, KL = {kl:.4f}")
print(f"identity check: H(p) + KL = {H_p + kl:.4f}")   # equals H(p,q)

Which direction? Mode-seeking vs mean-covering

Minimizing DKL(p∥q)D_{\mathrm{KL}}(p\|q) (forward, used by MLE) forces qq to cover everywhere pp has mass — spread out, or pay infinite penalty. Minimizing DKL(q∥p)D_{\mathrm{KL}}(q\|p) (reverse, used by variational inference and VAEs) lets qq lock onto one mode of pp and ignore the rest. Same formula, opposite personalities.

Steer the single-Gaussian model qq below, then let each button find its own best fit. Minimizing forward KL spreads qq wide to cover both peaks of the truth; minimizing reverse KL collapses qq onto the heavier peak and abandons the other.

Forward vs reverse KL: mode-covering vs mode-seeking

-201.3■ p (truth)— q (model)
H(p)1.982
cross-entropy H(p,q)2.443
forward KL(p‖q)0.461
reverse KL(q‖p)0.843

The truth p (grey) is bimodal; your model q (indigo) is a single Gaussian you steer with μ and σ. Forward KL(p‖q) blows up wherever p has mass but q doesn't, so its best fit spreads q to COVER both modes. Reverse KL(q‖p) blows up wherever q has mass but p doesn't, so its best fit LOCKS onto one mode and ignores the other. Same formula, opposite personalities.

Where you'll meet these again

  • Cross-entropy loss: every softmax classifier and language model (perplexity is just exponentiated cross-entropy, as in t-SNE's perplexity)
  • KL terms in objectives: the VAE's regularizer, policy-constraint terms in RLHF (keeping the fine-tuned model close to the base model)
  • Information gain: entropy reduction as a tree-split criterion

Exercise

Your model q assigns probability 0 to a word that actually occurs in the data distribution p. What happens to the forward KL divergence D(p‖q)?

Need a hint?

Look at the term p(x) · log(p(x)/q(x)) when q(x) = 0 but p(x) > 0.

Exercise

The cross-entropy H(p, q) always decomposes as which of the following?

Need a hint?

What's the 'extra surprise' term for using the wrong codebook?

Key takeaways

  • Entropy = expected surprise −∑plog⁡p-\sum p \log p: uniform maximizes it, certainty zeroes it
  • Cross-entropy = entropy + KL: training on log-loss minimizes the KL between data and model
  • KL is non-negative, zero only at equality, asymmetric, and infinite when the model says "impossible" about something real
  • Forward KL covers all modes; reverse KL picks one — the choice shapes generative models

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Related
Entropy & KL DivergenceProbability DistributionsDecision TreesMaximum Likelihood Estima…Naive BayesVariational Autoencoderst-SNE & UMAP