Chain Rule and Backpropagation
Backpropagation is not magic — it's the chain rule applied to computational graphs. It's what makes the loop's optimization stage tractable for deep nets: without an efficient way to get the gradient of the loss w.r.t. millions of parameters, gradient descent would be a non-starter. Once you understand the chain rule, every step of backprop becomes a mechanical application of one idea.
Real-world examples: Backpropagation (the algorithm that trains every deep neural network) is the chain rule applied recursively through a computational graph.
The chain rule
When a function is composed of other functions, the chain rule tells you how to differentiate the composition:
Example: where is sigmoid.
Let and :
The trick that powers backprop is to read this right to left. We never write the whole derivative as one giant formula. Instead each elementary operation contributes one local factor, and we multiply those factors together:
If a third operation were stacked on top, say , we would simply prepend one more factor: . Backprop is nothing more than this — a chain of single-operation derivatives multiplied together.
Multivariable chain rule
For where and :
When depends on and depends on via , the chain rule becomes a matrix product of Jacobians:
Computational graphs
A computational graph represents a function as a directed acyclic graph (DAG): nodes are operations, edges carry values forward (forward pass) and gradients backward (backward pass).
Backprop = the chain rule on a graph
Forward (teal) computes values left → right. Backward (orange) multiplies local derivatives right → left: dy/dx = dy/du · du/dx = cos(u) · 2x. Each node just multiplies the incoming gradient by its own local derivative.
Example graph:
x, w → [dot product] → z
z, y → [subtract] → r = z - y
r → [square] → L = r²
Forward pass
Compute each node's output in topological order:
Backward pass
Starting from , apply the chain rule in reverse:
Local gradients × upstream gradient
Each node stores its local gradient (derivative of output w.r.t. its inputs). During backprop, it multiplies the incoming upstream gradient by the local gradient and passes it back. The chain rule is always: local × upstream.
A fully worked numeric pass
Let's run the squared-error graph with real numbers so every multiplication is concrete. Take a single weight and feature , so (a one-element dot product), with
Forward pass — compute and cache each intermediate value:
Backward pass — start at the output with and walk back, multiplying upstream × local at each node:
So . We can sanity-check against the closed form . ✅ The notebook reproduces this and confirms it against a finite-difference estimate.
Worked example: one hidden layer
Now scale the exact same procedure up to a tiny MLP. Network:
Name every intermediate node so we can attach a gradient to each one:
The backward pass is one chain-rule multiplication per node, output → input. The first step uses a classic simplification: for binary cross-entropy composed with sigmoid, the two local gradients collapse to a single clean expression,
From there each subsequent line is upstream × local:
where is element-wise product and is where and elsewhere. Notice how the cached forward values (, , , ) are exactly what each local gradient needs — that reuse is why one backward pass suffices.
import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))
def relu(z): return np.maximum(0, z)
# Forward pass
np.random.seed(0) # reproducible
x = np.array([1.0, 2.0, 3.0])
W1 = np.random.randn(4, 3) * 0.1
b1 = np.zeros(4)
w2 = np.random.randn(4) * 0.1
b2 = 0.0
y = 1.0
z1 = W1 @ x + b1 # pre-activation layer 1
a1 = relu(z1) # activation layer 1
z2 = w2 @ a1 + b2 # pre-activation output
y_hat = sigmoid(z2) # predicted probability
L = -(y * np.log(y_hat) + (1-y) * np.log(1-y_hat))
# Backward pass — one line per node, upstream × local
dL_dz2 = y_hat - y # BCE + sigmoid collapse
dL_dw2 = dL_dz2 * a1 # × a1
dL_db2 = dL_dz2 # × 1
dL_da1 = dL_dz2 * w2 # × w2
dL_dz1 = dL_da1 * (z1 > 0) # × ReLU'(z1)
dL_dW1 = np.outer(dL_dz1, x) # × xᵀ
dL_db1 = dL_dz1 # × 1
Watch both passes run
The graph above shows the shape; this steps the numbers through it — first the scalar graph worked by hand, then the same procedure on this network, with every gradient checked against a finite difference at the end:
Backpropagation through a computational graph, checked against finite differences
Start with the scalar graph the lesson works by hand: L = (wx − y)² with w = 2, x = 3, y = 5. The forward pass walks the graph in topological order and — this is the part that matters — **caches every intermediate**: z = 6, r = 1, L = 1. Nothing is thrown away, because each of those values is about to be needed again on the way back.
| node | value | how |
|---|---|---|
| z | 6 | w · x = 2 · 3 |
| r | 1 | z − y = 6 − 5 |
| L | 1 | r² = 1² |
Every node stores its output. That cache is the entire reason a single backward pass can produce every gradient — the alternative is recomputing the forward values once per parameter.
The scalar graph worked by hand above (w = 2, x = 3, y = 5 → ∂L/∂w = 6), then the identical procedure on the one-hidden-layer network. The weights are small explicit constants so every number is checkable, and they put one hidden unit at z₁ = −0.1: watch that unit get switched off by the ReLU and take 5 of the network's 11 parameter gradients to exactly zero with it — while ∂L/∂a₁ for the same unit stays non-zero, which is the distinction the gate exists to draw. The last two frames verify every gradient against a central finite difference (worst relative error 3.9e-11) and count what that verification cost: 22 forward passes against backprop's one.
What to notice:
- The cache is the whole trick. Every local gradient is a function of a value the forward pass already computed — needs , needs , needs . Nothing is recomputed.
- The BCE + sigmoid collapse is exact. Separately the two factors are and ; their product is to the last digit. That is why frameworks fuse the two ops — the fused form also avoids dividing by .
- Non-zero , zero . The trace's second hidden unit sits at . The loss is genuinely sensitive to its activation (), but the ReLU gate zeroes the gradient to its pre-activation, and those are different quantities. This is the mechanism behind dying ReLU: the gradient that would revive the unit is precisely the one being switched off.
- One dead unit zeroes 5 of the 11 parameter gradients: the whole second row of , its bias, and the output weight reading from it.
- The efficiency claim, counted. Verifying those 11 gradients by finite differences takes 22 forward passes; backprop produces all of them in one backward pass. At parameters that ratio is .
Why backprop is efficient
Naively computing by finite differences requires two forward passes per parameter. A network with millions of parameters would be intractable.
Backpropagation computes all gradients in one backward pass by reusing intermediate results. The key insight: gradients at earlier layers are computed by multiplying already-computed downstream gradients with local Jacobians. Total cost: – the forward pass cost (one pass to compute activations, one to propagate gradients, plus the weight gradient accumulation).
What the framework does with this
You never write the backward pass by hand, because PyTorch and JAX implement reverse-mode automatic differentiation: they build the computational graph on the fly as your forward pass runs, storing each operation's local derivative alongside its output, then replay that record in reverse.
Two details of that implementation are worth knowing, because both leak into APIs you use daily. The graph is replayed in reverse topological order, so a node's gradient is never read until every operation that consumed it has contributed. And contributions accumulate with += rather than overwrite, which is what makes a reused parameter come out right: a weight shared across 50 RNN timesteps is just a node with 50 outgoing edges, and the accumulation is the multivariable chain rule's sum over paths. It is also why optimizer.zero_grad() is your job rather than the framework's.
Vanishing and exploding gradients
When many small Jacobians are multiplied together (deep networks, long sequences), the gradient can vanish:
If each , the gradient decays exponentially. Sigmoid's max derivative is — stacking 20 sigmoid layers multiplies gradients by at most .
Solutions:
- ReLU: gradient is exactly 1 for positive inputs, no saturation
- Batch normalization: stabilizes activations, preventing saturation
- Residual connections: add the input directly, creating gradient highways
Exercise
For h(x) = (3x + 1)⁴, what is h'(x)?
Need a hint?
Use the chain rule: d/dx f(g(x)) = f'(g(x)) · g'(x).
Exercise
In backpropagation, the gradient at each node is computed as the product of what two things?
Need a hint?
Each node has its own derivative and receives a signal from downstream.
Exercise
Why does stacking many sigmoid activations cause vanishing gradients?
Need a hint?
Think about what happens when you multiply many numbers less than 1.
Exercise
In a reverse-mode autodiff engine the backward pass ends with `parent.grad += local(node.grad)`. A colleague argues the `+=` should be `=`, since each node's gradient is computed exactly once. What breaks?
Need a hint?
What happens to a tensor that is used as an input by more than one operation?
Exercise
Why must a reverse-mode backward pass visit nodes in reverse topological order rather than any convenient order (say, the order they were created)?
Need a hint?
Think about a node whose output feeds two different downstream operations.
Exercise
During training, activation memory is usually the largest consumer of GPU memory — often larger than parameters, gradients and optimizer state combined. Which property of reverse-mode autodiff causes this?
Need a hint?
When is each operation's local derivative rule created, and what values does it need to close over?
Common mistakes
- Forgetting to cache forward-pass activations: backprop needs them to compute local gradients; without caching, you'd have to recompute the entire forward pass for every parameter.
- Confusing vanishing gradients with the chain rule being wrong: vanishing gradients are a consequence of multiplying many small terms (< 1) together, not an error in the chain rule itself.
- Applying the chain rule to matrix operations without respecting the transpose: involves , not ; transposing preserves dimension consistency in the gradient.
Key takeaways
- Backpropagation = chain rule applied to a computational graph.
- Each node multiplies the upstream gradient by its local gradient and passes it back.
- Backprop computes all gradients simultaneously in one pass. This is why neural network training is tractable.
- Vanishing gradients happen when many factors multiply; solved by ReLU, batch norm, or residual connections.
Related concepts
- Derivatives and Gradients: the building blocks
- Gradient Descent: using the gradients backprop computes
- Recurrent Neural Networks: backprop through time (BPTT)