Information Theory for ML: Entropy, KL Divergence, and Cross-Entropy
Shannon's information theory provides the unifying language behind probabilistic machine learning. The losses you minimize (cross-entropy, KL divergence, negative log-likelihood) are all information-theoretic quantities in disguise. This page derives them from first principles and shows where each appears in ML.
Surprisal: the atom of information
The surprisal (self-information) of an event with probability is:
Rare events carry more information than common ones: if (certain), — no surprise, no information. If , bits — a very surprising event.
Shannon entropy
Entropy is the expected surprisal of a distribution :
Entropy measures the average uncertainty or the minimum average bits needed to encode samples from .
Key values:
- : no uncertainty — one outcome has (deterministic).
- : maximum entropy — the uniform distribution over outcomes (maximum uncertainty).
- For a fair coin (): bit.
Worked example — biased coin with :
Less than 1 bit, because the outcome is partially predictable.
KL divergence: the cost of being wrong
The Kullback-Leibler divergence measures the extra bits needed to encode samples from using a code designed for instead of :
Properties:
- always, with equality iff (Gibbs inequality).
- Not symmetric: in general.
- Infinite if assigns zero probability to any event that does not.
Forward vs reverse KL — why the direction matters:
| Direction | Definition | Behavior | Used in |
|---|---|---|---|
| Forward | Samples from , penalizes where | Mean-seeking: must cover all modes of | VAE encoder, flow training |
| Reverse | Samples from , penalizes where | Mode-seeking: collapses onto one mode of | Variational inference, RL policy optimization |
In a multimodal , forward KL forces to spread mass across all modes (averaging). Reverse KL lets be a tight Gaussian locked onto one mode. Neither is "correct" — it depends on what you want the approximate posterior to do.
Cross-entropy: the training loss
Cross-entropy is the expected code length of -samples under a -code:
Since is fixed (the true data distribution), minimizing cross-entropy over the parameters of is equivalent to minimizing .
The ML training loss connection:
For a classifier with output distribution and true label :
where is the one-hot true distribution. Summing over training examples:
This is exactly negative log-likelihood (NLL). Minimizing cross-entropy = maximizing likelihood = minimizing KL between the empirical distribution and the model.
Every classification loss is a KL divergence
Binary cross-entropy (sigmoid + BCE), categorical cross-entropy (softmax + CE), and even mean squared error (Gaussian NLL) are all KL divergences between the empirical label distribution and the model's output distribution. The loss function you choose encodes an assumption about the noise distribution of your labels.
Mutual information: measuring statistical dependence
Mutual information measures how much knowing reduces uncertainty about :
- : and are independent — knowing tells you nothing about .
- : and are deterministically related.
- — symmetric (unlike KL divergence).
In ML:
- Feature selection: rank input features by — high MI features are informative.
- Information bottleneck: compress into a representation that retains as much (label info) as possible while discarding (irrelevant detail).
- Contrastive learning: maximizing MI between two views of the same example (CLIP, SimCLR) is equivalent to cross-entropy contrastive loss.
The chain connecting everything
Minimizing cross-entropy (cross-entropy loss in code) minimizes the KL divergence between the model and the data-generating distribution. The entropy floor sets the best achievable loss — a model can do no better than the irreducible randomness in the labels.
Numerical computation
Cross-entropy requires computing . When is produced by a softmax, numerical stability requires using log-sum-exp rather than computing softmax then log:
# Numerically stable: log-softmax + NLL loss
import torch.nn.functional as F
# y_pred: raw logits (N, C); y_true: class indices (N,)
loss = F.cross_entropy(y_pred, y_true) # = F.nll_loss(F.log_softmax(y_pred, dim=1), y_true)
For KL divergence between two distributions:
# p_log, q_log: log-probabilities
kl = F.kl_div(q_log, p_log, reduction='batchmean', log_target=True)
Related concepts
- Maximum Likelihood Estimation: MLE as NLL minimization
- Layers & the Forward Pass: cross-entropy as the output layer's loss
- Softmax and Cross-Entropy: the numerically stable log-softmax derivation
- Probability Distributions: the distributions entropy and KL operate on
- Bayesian Inference: KL divergence in variational inference (ELBO)