Information Theory for ML: Entropy, KL Divergence, and Cross-EntropyOpen in Colab

Information Theory for ML: Entropy, KL Divergence, and Cross-Entropy

Shannon's information theory provides the unifying language behind probabilistic machine learning. The losses you minimize (cross-entropy, KL divergence, negative log-likelihood) are all information-theoretic quantities in disguise. This page derives them from first principles and shows where each appears in ML.

Surprisal: the atom of information

The surprisal (self-information) of an event with probability pp is:

I(x)=−log⁡2p(x)(bits)orI(x)=−ln⁡p(x)(nats)I(x) = -\log_2 p(x) \quad \text{(bits)} \qquad \text{or} \qquad I(x) = -\ln p(x) \quad \text{(nats)}

Rare events carry more information than common ones: if p(x)=1p(x) = 1 (certain), I(x)=0I(x) = 0 — no surprise, no information. If p(x)=0.001p(x) = 0.001, I(x)≈10I(x) \approx 10 bits — a very surprising event.

Shannon entropy

Entropy is the expected surprisal of a distribution PP:

H(P)=Ex∼P[−log⁡p(x)]=−∑xp(x)log⁡p(x)H(P) = \mathbb{E}_{x \sim P}[-\log p(x)] = -\sum_x p(x) \log p(x)

Entropy measures the average uncertainty or the minimum average bits needed to encode samples from PP.

Key values:

  • H=0H = 0: no uncertainty — one outcome has p=1p = 1 (deterministic).
  • H=log⁡KH = \log K: maximum entropy — the uniform distribution over KK outcomes (maximum uncertainty).
  • For a fair coin (p=0.5p = 0.5): H=−0.5log⁡0.5−0.5log⁡0.5=1H = -0.5\log 0.5 - 0.5\log 0.5 = 1 bit.

Worked example — biased coin with p=0.8p = 0.8:

H=−0.8log⁡20.8−0.2log⁡20.2=0.8×0.322+0.2×2.322≈0.722 bitsH = -0.8\log_2 0.8 - 0.2\log_2 0.2 = 0.8 \times 0.322 + 0.2 \times 2.322 \approx 0.722 \text{ bits}

Less than 1 bit, because the outcome is partially predictable.

KL divergence: the cost of being wrong

The Kullback-Leibler divergence measures the extra bits needed to encode samples from PP using a code designed for QQ instead of PP:

DKL(P∥Q)=∑xp(x)log⁡p(x)q(x)=Ex∼P ⁣[log⁡p(x)q(x)]D_\text{KL}(P \| Q) = \sum_x p(x) \log \frac{p(x)}{q(x)} = \mathbb{E}_{x \sim P}\!\left[\log \frac{p(x)}{q(x)}\right]

Properties:

  • DKL(P∥Q)≥0D_\text{KL}(P \| Q) \ge 0 always, with equality iff P=QP = Q (Gibbs inequality).
  • Not symmetric: DKL(P∥Q)≠DKL(Q∥P)D_\text{KL}(P \| Q) \ne D_\text{KL}(Q \| P) in general.
  • Infinite if QQ assigns zero probability to any event that PP does not.

Forward vs reverse KL — why the direction matters:

DirectionDefinitionBehaviorUsed in
Forward DKL(P∥Q)D_\text{KL}(P \| Q)Samples from PP, penalizes Q≈0Q \approx 0 where P>0P > 0Mean-seeking: QQ must cover all modes of PPVAE encoder, flow training
Reverse DKL(Q∥P)D_\text{KL}(Q \| P)Samples from QQ, penalizes Q>0Q > 0 where P≈0P \approx 0Mode-seeking: QQ collapses onto one mode of PPVariational inference, RL policy optimization

In a multimodal PP, forward KL forces QQ to spread mass across all modes (averaging). Reverse KL lets QQ be a tight Gaussian locked onto one mode. Neither is "correct" — it depends on what you want the approximate posterior to do.

Cross-entropy: the training loss

Cross-entropy is the expected code length of PP-samples under a QQ-code:

H(P,Q)=−∑xp(x)log⁡q(x)=H(P)+DKL(P∥Q)H(P, Q) = -\sum_x p(x) \log q(x) = H(P) + D_\text{KL}(P \| Q)

Since H(P)H(P) is fixed (the true data distribution), minimizing cross-entropy over the parameters of QQ is equivalent to minimizing DKL(P∥Q)D_\text{KL}(P \| Q).

The ML training loss connection:

For a classifier with output distribution p^(y∣x)\hat{p}(y \mid x) and true label y∗y^*:

LCE=−log⁡p^(y∗∣x)=H(δy∗, p^)\mathcal{L}_\text{CE} = -\log \hat{p}(y^* \mid x) = H(\delta_{y^*},\, \hat{p})

where δy∗\delta_{y^*} is the one-hot true distribution. Summing over training examples:

L=−1n∑i=1nlog⁡p^(yi∣xi)\mathcal{L} = -\frac{1}{n}\sum_{i=1}^n \log \hat{p}(y_i \mid x_i)

This is exactly negative log-likelihood (NLL). Minimizing cross-entropy = maximizing likelihood = minimizing KL between the empirical distribution and the model.

Every classification loss is a KL divergence

Binary cross-entropy (sigmoid + BCE), categorical cross-entropy (softmax + CE), and even mean squared error (Gaussian NLL) are all KL divergences between the empirical label distribution and the model's output distribution. The loss function you choose encodes an assumption about the noise distribution of your labels.

Mutual information: measuring statistical dependence

Mutual information measures how much knowing XX reduces uncertainty about YY:

I(X;Y)=H(X)−H(X∣Y)=DKL(P(X,Y)∥P(X)P(Y))I(X; Y) = H(X) - H(X \mid Y) = D_\text{KL}(P(X,Y) \| P(X)P(Y))

  • I(X;Y)=0I(X;Y) = 0: XX and YY are independent — knowing XX tells you nothing about YY.
  • I(X;Y)=H(X)=H(Y)I(X;Y) = H(X) = H(Y): XX and YY are deterministically related.
  • I(X;Y)=I(Y;X)I(X;Y) = I(Y;X) — symmetric (unlike KL divergence).

In ML:

  • Feature selection: rank input features by I(feature;label)I(\text{feature}; \text{label}) — high MI features are informative.
  • Information bottleneck: compress XX into a representation ZZ that retains as much I(Z;Y)I(Z;Y) (label info) as possible while discarding I(Z;X)−I(Z;Y)I(Z;X) - I(Z;Y) (irrelevant detail).
  • Contrastive learning: maximizing MI between two views of the same example (CLIP, SimCLR) is equivalent to cross-entropy contrastive loss.

The chain connecting everything

H(P,Q)⏟cross-entropy loss=H(P)⏟irreducible entropy+DKL(P∥Q)⏟what the model controls\underbrace{H(P, Q)}_{\text{cross-entropy loss}} = \underbrace{H(P)}_{\text{irreducible entropy}} + \underbrace{D_\text{KL}(P \| Q)}_{\text{what the model controls}}

Minimizing cross-entropy (cross-entropy loss in code) minimizes the KL divergence between the model and the data-generating distribution. The entropy floor H(P)H(P) sets the best achievable loss — a model can do no better than the irreducible randomness in the labels.

Numerical computation

Cross-entropy requires computing log⁡p^\log \hat{p}. When p^\hat{p} is produced by a softmax, numerical stability requires using log-sum-exp rather than computing softmax then log:

Python
# Numerically stable: log-softmax + NLL loss
import torch.nn.functional as F

# y_pred: raw logits (N, C); y_true: class indices (N,)
loss = F.cross_entropy(y_pred, y_true)   # = F.nll_loss(F.log_softmax(y_pred, dim=1), y_true)

For KL divergence between two distributions:

Python
# p_log, q_log: log-probabilities
kl = F.kl_div(q_log, p_log, reduction='batchmean', log_target=True)

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Information Theory for ML: Entrop…Softmax + Cross-Entropy G…Probability Distributions…Maximum Likelihood Estima…Layers & the Forward PassBayesian Inference