Softmax + Cross-Entropy GradientOpen in Colab

Softmax + Cross-Entropy Gradient

Softmax converts raw output scores (logits) into a valid probability distribution. Paired with cross-entropy loss, the gradient takes a remarkably clean form that every neural network practitioner should know cold.

The softmax formula and its properties

For KK classes with logits z=(z1,…,zK)\mathbf{z} = (z_1, \ldots, z_K):

p^k=ezk∑j=1Kezj,k=1,…,K\hat{p}_k = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}, \qquad k = 1, \ldots, K

Properties:

  • All outputs positive: p^k>0\hat{p}_k > 0
  • Sum to one: ∑kp^k=1\sum_k \hat{p}_k = 1
  • Invariant to a constant offset: p^k(z+c)=p^k(z)\hat{p}_k(\mathbf{z} + c) = \hat{p}_k(\mathbf{z}) for any cc
  • Differences between logits determine confidence, not absolute values

A concrete 3-class example

Logits z=(2,1,−1)\mathbf{z} = (2, 1, -1):

ez≈(7.389, 2.718, 0.368),sum≈10.475e^{\mathbf{z}} \approx (7.389,\ 2.718,\ 0.368), \qquad \text{sum} \approx 10.475 p^≈(0.705, 0.259, 0.035)\hat{\mathbf{p}} \approx (0.705,\ 0.259,\ 0.035)

The model is 70.5% confident in class 1.

Cross-entropy loss

For a single example with true class yy (integer index):

L=−log⁡p^y=−log⁡ezy∑kezk=−zy+log⁡∑kezk\mathcal{L} = -\log \hat{p}_y = -\log \frac{e^{z_y}}{\sum_k e^{z_k}} = -z_y + \log\sum_k e^{z_k}

The log-sum-exp form is used in practice because it is numerically stable.

Numeric example (y=0y = 0 is correct): L=−log⁡(0.705)≈0.350\mathcal{L} = -\log(0.705) \approx 0.350.

Deriving the softmax + cross-entropy gradient

We want ∂L∂zk\frac{\partial \mathcal{L}}{\partial z_k} for each logit zkz_k.

For the correct class k=yk = y:

∂L∂zy=∂∂zy ⁣(−zy+log⁡∑jezj)=−1+ezy∑jezj=p^y−1\frac{\partial \mathcal{L}}{\partial z_y} = \frac{\partial}{\partial z_y}\!\left(-z_y + \log\sum_j e^{z_j}\right) = -1 + \frac{e^{z_y}}{\sum_j e^{z_j}} = \hat{p}_y - 1

For wrong classes k≠yk \neq y:

∂L∂zk=0+ezk∑jezj=p^k\frac{\partial \mathcal{L}}{\partial z_k} = 0 + \frac{e^{z_k}}{\sum_j e^{z_j}} = \hat{p}_k

Both cases unify:

∂L∂zk=p^k−1[k=y]\boxed{\frac{\partial \mathcal{L}}{\partial z_k} = \hat{p}_k - \mathbb{1}[k = y]}

Interpretation:

  • Correct class (k=yk=y): gradient =p^y−1≤0= \hat{p}_y - 1 \le 0 — gradient descent pushes logit up until p^y→1\hat{p}_y \to 1
  • Wrong classes: gradient =p^k≥0= \hat{p}_k \ge 0 — pushes logits down
  • Gradients sum to zero: ∑k(p^k−1[k=y])=1−1=0\sum_k (\hat{p}_k - \mathbb{1}[k=y]) = 1 - 1 = 0 — consistent with the constraint ∑kp^k=1\sum_k \hat{p}_k = 1

Verifying the gradient numerically

Python
import numpy as np

def softmax(z):
    e = np.exp(z - z.max())   # subtract max for numerical stability
    return e / e.sum()

def cross_entropy_loss(z, y):
    return -np.log(softmax(z)[y])

def softmax_ce_grad(z, y):
    grad = softmax(z).copy()
    grad[y] -= 1
    return grad

z = np.array([2.0, 1.0, -1.0])
y = 0

print(f"softmax: {softmax(z).round(4)}")
print(f"loss:    {cross_entropy_loss(z, y):.4f}")
print(f"grad:    {softmax_ce_grad(z, y).round(4)}")

# Finite-difference verification
eps = 1e-5
fd = np.zeros(len(z))
for i in range(len(z)):
    z_p, z_m = z.copy(), z.copy()
    z_p[i] += eps; z_m[i] -= eps
    fd[i] = (cross_entropy_loss(z_p, y) - cross_entropy_loss(z_m, y)) / (2 * eps)

print(f"FD:      {fd.round(4)}")
print(f"Match:   {np.allclose(softmax_ce_grad(z, y), fd, atol=1e-7)}")

The log-sum-exp numerical-stability trick

Direct computation of ezke^{z_k} overflows for large logits (e1000e^{1000} is inf in float32). Subtract the maximum before exponentiating — the offset cancels in the softmax ratio:

log⁡∑kezk=zmax⁡+log⁡∑kezk−zmax⁡\log\sum_k e^{z_k} = z_{\max} + \log\sum_k e^{z_k - z_{\max}}

Since zk−zmax⁡≤0z_k - z_{\max} \le 0, the exponents never overflow, and the value e0=1e^0 = 1 (the largest term) anchors the computation.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Softmax + Cross-Entropy GradientActivation Functions: Fro…Information Theory for ML…Logistic Regression: Grad…The EM AlgorithmXOR, MLPs & Softmax