Softmax + Cross-Entropy Gradient
Softmax converts raw output scores (logits) into a valid probability distribution. Paired with cross-entropy loss, the gradient takes a remarkably clean form that every neural network practitioner should know cold.
The softmax formula and its properties
For classes with logits :
Properties:
- All outputs positive:
- Sum to one:
- Invariant to a constant offset: for any
- Differences between logits determine confidence, not absolute values
A concrete 3-class example
Logits :
The model is 70.5% confident in class 1.
Cross-entropy loss
For a single example with true class (integer index):
The log-sum-exp form is used in practice because it is numerically stable.
Numeric example ( is correct): .
Deriving the softmax + cross-entropy gradient
We want for each logit .
For the correct class :
For wrong classes :
Both cases unify:
Interpretation:
- Correct class (): gradient — gradient descent pushes logit up until
- Wrong classes: gradient — pushes logits down
- Gradients sum to zero: — consistent with the constraint
Verifying the gradient numerically
import numpy as np
def softmax(z):
e = np.exp(z - z.max()) # subtract max for numerical stability
return e / e.sum()
def cross_entropy_loss(z, y):
return -np.log(softmax(z)[y])
def softmax_ce_grad(z, y):
grad = softmax(z).copy()
grad[y] -= 1
return grad
z = np.array([2.0, 1.0, -1.0])
y = 0
print(f"softmax: {softmax(z).round(4)}")
print(f"loss: {cross_entropy_loss(z, y):.4f}")
print(f"grad: {softmax_ce_grad(z, y).round(4)}")
# Finite-difference verification
eps = 1e-5
fd = np.zeros(len(z))
for i in range(len(z)):
z_p, z_m = z.copy(), z.copy()
z_p[i] += eps; z_m[i] -= eps
fd[i] = (cross_entropy_loss(z_p, y) - cross_entropy_loss(z_m, y)) / (2 * eps)
print(f"FD: {fd.round(4)}")
print(f"Match: {np.allclose(softmax_ce_grad(z, y), fd, atol=1e-7)}")
The log-sum-exp numerical-stability trick
Direct computation of overflows for large logits ( is inf in float32). Subtract the maximum before exponentiating — the offset cancels in the softmax ratio:
Since , the exponents never overflow, and the value (the largest term) anchors the computation.
Related concepts
- XOR, MLPs & Softmax: the full lesson: softmax properties, temperature scaling, multi-class classification
- Logistic Regression: Gradient Derivation: the binary case: sigmoid + BCE gives the same clean form
- The EM Algorithm: softmax probabilities appear as responsibilities in the E-step