Gradient Checkpointing
Backpropagation needs every layer's intermediate activations to compute gradients — the standard implementation caches all of them during the forward pass and consumes them during the backward pass. For a network with layers, that's activation memory that grows linearly in depth, and for today's deep models it is often the single biggest consumer of GPU memory, larger than the parameters or optimizer state. Gradient checkpointing (a.k.a. activation checkpointing, or "rematerialization") breaks this by storing only a sparse set of activations (the checkpoints) and recomputing the discarded ones on the fly during the backward pass. It trades extra forward compute for a memory footprint that grows only as .
1: Why plain backprop is in memory
For a feed-forward network with layers and activations (input) through (output):
The backward pass needs to compute and at every layer , via the chain rule. The straightforward implementation keeps all activations alive simultaneously — from the moment they're produced in the forward pass until the backward pass consumes them in reverse order:
2: The checkpoint-and-recompute idea
Partition the layers into segments of size , and during the forward pass, only store the activation at the start of each segment (the checkpoint) — discard everything computed inside a segment once it has been passed forward.
During the backward pass, process segments in reverse order. To get the gradients for the layers inside a segment, first replay a local forward pass from that segment's checkpoint to regenerate its intermediate activations, then run the ordinary backward pass through that freshly recomputed segment. Once that segment's gradients are obtained, discard the recomputed activations again and move to the previous segment.
- Forward pass: run as usual, but only retain (one activation per segment boundary). Discard every intra-segment activation immediately after it has been consumed by the next layer.
- Backward pass, per segment (last segment first): load the segment's checkpoint (its input activation), and re-run the forward computation only across that segment to reconstruct its intra-segment activations.
- Run ordinary backprop through the freshly recomputed segment, accumulating parameter gradients and the gradient flowing into the segment's input.
- Discard the recomputed activations (they've done their job) and move to the previous segment, chaining in the incoming gradient.
- Repeat until segment 1, arriving back at .
What's actually being traded
Gradient checkpointing doesn't reduce the total amount of forward computation ever performed in an absolute sense — it performs it twice for the layers inside each segment (once during the real forward pass, once during recomputation). What it eliminates is the need to hold every layer's activation in memory at once. That's the trade: a bounded, fixed increase in FLOPs for an asymptotic reduction in peak memory, which is what lets a model that would otherwise OOM actually fit on the GPU.
3: Choosing the segment size: why
Peak memory during the backward pass is the sum of two pools: the checkpoints still held for segments not yet processed, and the intra-segment activations currently being recomputed. With segment size (so there are checkpoints) and up to activations live inside whichever segment is currently being replayed:
Minimize over by setting the derivative to zero:
Substituting back gives the optimal peak memory:
This is the standard heuristic: checkpoint every layers. Too few checkpoints (large ) means each recompute is expensive in memory; too many checkpoints (small ) means the checkpoints themselves dominate memory. balances the two pools exactly.
4: Why the extra compute is only ~30–40%, not 2×
It's tempting to assume checkpointing roughly doubles training cost, since every checkpointed layer's forward is repeated. But the recomputation only replays the forward pass — it never repeats the backward pass, which is the more expensive half. Using the standard rule of thumb that a full training step costs about 1 unit of forward FLOPs and about 2 units of backward FLOPs (backward touches both the input and weight gradients):
The recomputed segments sum to exactly one extra full forward pass over the whole network (each layer is recomputed exactly once, regardless of how many segments it's split into), so the ~30–40% overhead is roughly constant — it doesn't get worse as grows, while the memory savings keep improving.
Worked example: toy -layer MLP
Take a toy MLP where each layer's activation costs 1 "unit" of memory, and compare no-checkpointing against checkpointing with segment size :
| (layers) | (segment size) | checkpoints stored | peak memory (checkpoints + current segment) | memory: no-checkpoint | reduction factor | extra forward compute |
|---|---|---|---|---|---|---|
| 16 | 4 | 4 | 8 | 16 | 2× | ~33% |
| 64 | 8 | 8 | 16 | 64 | 4× | ~33% |
| 256 | 16 | 16 | 32 | 256 | 8× | ~33% |
| 1024 | 32 | 32 | 64 | 1024 | 16× | ~33% |
Reading the row concretely: without checkpointing you hold all 64 layer activations simultaneously (64 units). With checkpointing every layers, you store 8 checkpoints (at layers ); when the backward pass reaches the segment for layers –, it replays that segment's forward pass, transiently holding up to 8 more activations (a peak of units, a reduction) for a fixed ~33% increase in forward FLOPs. As grows the memory reduction factor grows like , while the compute tax stays flat: exactly the trade that makes checkpointing more valuable the deeper the model gets.
Related concepts
- GPUs for Deep Learning: mixed precision is the sibling memory-saving technique (halves activation size); checkpointing reduces activation count instead
- MLOps Infrastructure & Orchestration: the resource-management trade-offs that make techniques like checkpointing part of a production training stack