Automatic Differentiation: How Autograd Actually Works
Most explanations of backpropagation stop at the chain rule, which is the mathematics. This page is about the implementation: the data structure PyTorch, JAX and TensorFlow build while your forward pass runs, and the algorithm they run over it when you call .backward(). It sits in the optimization slot of the ML loop, and what it replaced was writing derivatives by hand for every architecture you ever wanted to try.
The gap is worth naming, because "backprop is just the chain rule" leaves you unable to answer the questions that actually come up: why activations dominate your memory bill, why zero_grad() exists, why a tensor used twice needs no special handling, and what retain_graph=True is retaining.
Four ways to get a derivative
Autodiff is neither symbolic differentiation nor numerical differentiation, and the distinction is the entire reason it works at scale.
| Approach | How | Cost for , | Problem |
|---|---|---|---|
| Symbolic | Manipulate the expression algebraically | — | Expression swell: the derivative of a deep composition is exponentially large to write down, and control flow has no symbolic form |
| Numerical | per input | forward passes | Too slow (), and truncation vs. round-off means you never get more than ~8 digits |
| Forward-mode AD | Propagate derivatives alongside values | forward passes | Cost scales with inputs |
| Reverse-mode AD | Record values forward, propagate derivatives backward | 1 forward + 1 backward | Must store the intermediates |
The last row is the one deep learning runs on. The cost of reverse mode is independent of the number of parameters, so a single backward pass produces gradients for all of them at roughly 2× the cost of the forward pass.
The rule for choosing a mode
For , forward mode costs passes and
reverse mode costs . Training has = millions of parameters and
(a scalar loss), so reverse mode wins by a factor of . Flip the
shape — one input, many outputs, as in sensitivity analysis or Jacobian-vector
products — and forward mode is the right tool. JAX exposes both directly as
jax.grad / jax.jvp, which is why its API distinguishes jvp from vjp
where PyTorch mostly just says backward.
The tape
A framework never sees your formula. It sees a sequence of operation calls, and it writes each one down as it happens. That record — the tape, or the autograd graph — is built as a side effect of the forward pass.
Each recorded node stores three things:
- The output value, which the next operation needs.
- References to its input nodes, which are the edges of the graph.
- A closure that turns an incoming gradient into gradients for those inputs — the local derivative, with whatever forward values it needs captured at record time.
Point 3 is where activation memory comes from. mul(a, b) has to remember a and b to compute its backward, because . Every op makes its own choice about what to keep: tanh keeps its output (since is cheapest from ), ReLU keeps only a sign mask, matmul keeps both operands. Those decisions, summed over every layer, are usually a larger number than your parameter count.
Reverse-mode autodiff: building the tape and replaying it backwards
The expression, and why this one. a = w·x, h = tanh(a), o = w·h, d = o − t, L = d·d, with w = 0.7, x = 1.5, t = 1. Look at where **w** appears: once at the very bottom feeding the tanh, and again near the top multiplying its output. One parameter, two uses. That is weight sharing — the same structure an RNN has across timesteps — and it is precisely the case where "just apply the chain rule" stops being a complete instruction, because w's gradient is a sum over two paths of different lengths.
A framework does not read your expression. It has no idea a formula exists — it only sees a sequence of operations being called, and it writes each one down as it happens. That written record is the tape.
A real micro-autograd, recorded as it runs. The expression uses w twice (a = w·x and o = w·h) so it is a genuine DAG rather than a chain, which is what makes the two structural pieces necessary: the topological sort, so a node's gradient is never read before all of its consumers have written to it, and the `+=`, so both paths add instead of one overwriting the other. The last frame changes that `+=` to `=`, re-runs the traversal, and checks both against a finite difference — the bug returns 17.2% of the true gradient with no error of any kind, and the frame breaks down why that is even worse than w's own two paths would explain.
What to notice as you step through it:
- The local derivatives are stored during the forward pass, not computed during the backward one. The backward pass is a replay, not a derivation.
wis used twice, and nothing in the recording treats that as special. The tape just gets a second edge into the same node.- The topological sort is what makes the second edge safe. A node's gradient is not final until every consumer has pushed into it.
parent.g += ...is load-bearing. The final frame changes it to=and gets 34.3% of the correct gradient back, with no error raised.
Why the order matters
The backward pass cannot run in any order. Consider the fan-out from the trace: w feeds both a = w·x and o = w·h. Its true gradient is a sum over paths:
If you read w.grad after the short path has contributed but before the long one has, you get a number that is well-formed, plausible, and wrong. The invariant a correct implementation must maintain is:
Never process a node until every node that consumes its output has already been processed.
That is exactly reverse topological order. PyTorch enforces it with a dependency counter — each node knows how many consumers still owe it a contribution, and enters the ready queue when that count hits zero — rather than a depth-first sort, but the guarantee is the same.
The multivariable chain rule is where the sum comes from: for depending on through several intermediates ,
Accumulation is that sum, implemented. This is why weight sharing needs no special support: an RNN reusing across 50 timesteps, or a convolution reusing a kernel across every spatial position, is just a node with 50 (or 50,000) outgoing edges, and += handles it identically to two.
Define-by-run vs. define-and-run
Two designs, and the trade-off between them explains most of the last decade of framework churn.
| Define-by-run (dynamic) | Define-and-run (static) | |
|---|---|---|
| When the graph is built | While the forward pass executes, every iteration | Once, ahead of time |
| Examples | PyTorch eager, TF2 eager, Chainer (the original) | TensorFlow 1.x, torch.compile, jax.jit, ONNX |
| Control flow | Ordinary Python if/while — the tape records whichever branch ran | Needs graph-level primitives (tf.cond, lax.scan) or tracing that bakes one branch in |
| Debugging | Standard stack traces and breakpoints | Errors surface at graph-execution time, far from the code that caused them |
| Optimization | Limited: the framework sees one op at a time | Whole-graph: operator fusion, memory planning, dead-code elimination, kernel selection |
Dynamic won on ergonomics, and then the industry got the optimization back by tracing dynamic code into a static graph (torch.compile, jax.jit). That is also where tracing's sharp edge comes from: a traced function records the branch taken for the shapes and values it was traced with. Python control flow that depends on tensor values is silently frozen into the trace, which is why JAX makes you write lax.cond and why torch.compile inserts guards and re-compiles when they fail.
What it costs, and what the API exposes
Every piece of autograd's bookkeeping shows up somewhere in the API you already use.
- Activation memory. The tape holds forward values alive until the backward pass consumes them, so memory scales with depth × batch × width, not with parameter count. This is what trades away, storing a sparse set of checkpoints and recomputing the rest.Deep diveGradient CheckpointingIncludes runnable notebook
loss.backward()frees the graph by default. Once the buffers have been consumed they are released, which is why a secondbackward()on the same graph raises an error and whyretain_graph=Trueexists. You need it when one forward pass feeds two losses backwarded separately, as in some GAN and meta-learning setups.optimizer.zero_grad()is required because of+=. The framework's accumulation never resets on its own: it cannot know where one backward pass ends and the next begins. That is a feature — it is how you implement gradient accumulation across micro-batches to simulate a larger batch — but forget it and you are summing this step's gradient onto last step's.torch.no_grad()and.detach()stop the recording. The first suppresses tape-building entirely (inference, and the optimizer's own parameter updates); the second cuts one tensor out of the graph, which is how you stop a gradient flowing into a target network or a frozen encoder.- Only leaf tensors keep
.grad. Intermediates get their gradients passed along and then dropped, because keeping them all would double an already-large memory bill.retain_grad()opts a specific intermediate back in. - In-place operations can corrupt the backward pass. If an op stored a tensor for its backward and you overwrite that tensor in place, the replay uses the wrong number. PyTorch version-counts tensors and raises rather than returning a silently wrong gradient.
Autodiff is exact, but only of the code you wrote
Reverse-mode AD is not an approximation — it applies exact derivative rules,
accurate to floating-point round-off, which is why it beats finite differences
on accuracy as well as speed. But it differentiates the program that ran,
and where your program is not differentiable it will still hand you a number.
relu at exactly 0, abs at 0, and every round/floor/argmax returns a
convenient subgradient (usually 0) with no warning. Straight-through estimators
and the Gumbel-softmax trick exist precisely to put a useful fiction there
instead of the default one.
Building intuition: implement it once
The notebook for this page builds a working reverse-mode autodiff engine in about 60 lines of NumPy — Value objects, operator overloading, topological sort, accumulation — trains a small MLP with it, and checks every gradient against torch.autograd. It is the single most useful hour available for making frameworks stop feeling like magic, and it is also the fastest way to internalize why the answer to "what happens in the backward pass?" is a data-structure answer rather than a calculus one.
Related concepts
- Chain Rule and Backpropagation: the mathematics this page implements
- Jacobians & Vector Calculus: VJPs, which are what each node's backward closure actually computes
- Gradient Checkpointing: trading recomputation for the activation memory the tape holds
- Backpropagation Through Time: the fan-out case at scale, one edge per timestep
- Vanishing Gradient Analysis: what the repeated multiplication in the backward pass does to deep graphs