Gradient CheckpointingOpen in Colab

Gradient Checkpointing

Backpropagation needs every layer's intermediate activations to compute gradients — the standard implementation caches all of them during the forward pass and consumes them during the backward pass. For a network with LL layers, that's activation memory that grows linearly in depth, and for today's deep models it is often the single biggest consumer of GPU memory, larger than the parameters or optimizer state. Gradient checkpointing (a.k.a. activation checkpointing, or "rematerialization") breaks this by storing only a sparse set of activations (the checkpoints) and recomputing the discarded ones on the fly during the backward pass. It trades extra forward compute for a memory footprint that grows only as O(L)O(\sqrt{L}).

1: Why plain backprop is O(L)O(L) in memory

For a feed-forward network with layers f1,…,fLf_1, \dots, f_L and activations a0a_0 (input) through aLa_L (output):

ai=fi(ai−1),i=1,…,La_i = f_i(a_{i-1}), \qquad i = 1, \dots, L

The backward pass needs ai−1a_{i-1} to compute ∂L/∂θi\partial \mathcal{L}/\partial \theta_i and ∂L/∂ai−1\partial \mathcal{L}/\partial a_{i-1} at every layer ii, via the chain rule. The straightforward implementation keeps all L+1L{+}1 activations alive simultaneously — from the moment they're produced in the forward pass until the backward pass consumes them in reverse order:

memoryno-checkpoint=O(L)\text{memory}_{\text{no-checkpoint}} = O(L)

2: The checkpoint-and-recompute idea

Partition the LL layers into segments of size mm, and during the forward pass, only store the activation at the start of each segment (the checkpoint) — discard everything computed inside a segment once it has been passed forward.

During the backward pass, process segments in reverse order. To get the gradients for the layers inside a segment, first replay a local forward pass from that segment's checkpoint to regenerate its intermediate activations, then run the ordinary backward pass through that freshly recomputed segment. Once that segment's gradients are obtained, discard the recomputed activations again and move to the previous segment.

  1. Forward pass: run a0→aLa_0 \to a_L as usual, but only retain a0,am,a2m,a3m,…a_0, a_m, a_{2m}, a_{3m}, \dots (one activation per segment boundary). Discard every intra-segment activation immediately after it has been consumed by the next layer.
  2. Backward pass, per segment (last segment first): load the segment's checkpoint (its input activation), and re-run the forward computation only across that segment to reconstruct its intra-segment activations.
  3. Run ordinary backprop through the freshly recomputed segment, accumulating parameter gradients and the gradient flowing into the segment's input.
  4. Discard the recomputed activations (they've done their job) and move to the previous segment, chaining in the incoming gradient.
  5. Repeat until segment 1, arriving back at a0a_0.

What's actually being traded

Gradient checkpointing doesn't reduce the total amount of forward computation ever performed in an absolute sense — it performs it twice for the layers inside each segment (once during the real forward pass, once during recomputation). What it eliminates is the need to hold every layer's activation in memory at once. That's the trade: a bounded, fixed increase in FLOPs for an asymptotic reduction in peak memory, which is what lets a model that would otherwise OOM actually fit on the GPU.

3: Choosing the segment size: why L\sqrt{L}

Peak memory during the backward pass is the sum of two pools: the checkpoints still held for segments not yet processed, and the intra-segment activations currently being recomputed. With segment size mm (so there are L/mL/m checkpoints) and up to mm activations live inside whichever segment is currently being replayed:

peak memory(m)  ≈  Lm⏟checkpoints+m⏟current segment\text{peak memory}(m) \;\approx\; \underbrace{\frac{L}{m}}_{\text{checkpoints}} + \underbrace{m}_{\text{current segment}}

Minimize over mm by setting the derivative to zero:

ddm(Lm+m)=−Lm2+1=0⟹m=L\frac{d}{dm}\left(\frac{L}{m} + m\right) = -\frac{L}{m^2} + 1 = 0 \quad\Longrightarrow\quad m = \sqrt{L}

Substituting back gives the optimal peak memory:

peak memory(L)=LL+L=2L=O(L)\text{peak memory}(\sqrt{L}) = \frac{L}{\sqrt{L}} + \sqrt{L} = 2\sqrt{L} = O(\sqrt{L})

This is the standard heuristic: checkpoint every L\sqrt{L} layers. Too few checkpoints (large mm) means each recompute is expensive in memory; too many checkpoints (small mm) means the checkpoints themselves dominate memory. m=Lm = \sqrt{L} balances the two pools exactly.

4: Why the extra compute is only ~30–40%, not 2×

It's tempting to assume checkpointing roughly doubles training cost, since every checkpointed layer's forward is repeated. But the recomputation only replays the forward pass — it never repeats the backward pass, which is the more expensive half. Using the standard rule of thumb that a full training step costs about 1 unit of forward FLOPs and about 2 units of backward FLOPs (backward touches both the input and weight gradients):

no checkpointing: 1 (fwd)+2 (bwd)=3 units\text{no checkpointing: } 1\,(\text{fwd}) + 2\,(\text{bwd}) = 3 \text{ units} with checkpointing: 1 (fwd)+1 (recompute fwd)+2 (bwd)=4 units\text{with checkpointing: } 1\,(\text{fwd}) + 1\,(\text{recompute fwd}) + 2\,(\text{bwd}) = 4 \text{ units} extra compute=4−33≈33%\text{extra compute} = \frac{4-3}{3} \approx 33\%

The recomputed segments sum to exactly one extra full forward pass over the whole network (each layer is recomputed exactly once, regardless of how many segments it's split into), so the ~30–40% overhead is roughly constant — it doesn't get worse as LL grows, while the memory savings keep improving.

Worked example: toy LL-layer MLP

Take a toy MLP where each layer's activation costs 1 "unit" of memory, and compare no-checkpointing against checkpointing with segment size m=Lm = \sqrt{L}:

LL (layers)L\sqrt{L} (segment size)checkpoints storedpeak memory (checkpoints + current segment)memory: no-checkpointreduction factorextra forward compute
16448162×~33%
648816644×~33%
2561616322568×~33%
1024323264102416×~33%

Reading the L=64L=64 row concretely: without checkpointing you hold all 64 layer activations simultaneously (64 units). With checkpointing every 88 layers, you store 8 checkpoints (at layers 0,8,16,…,560, 8, 16, \dots, 56); when the backward pass reaches the segment for layers 4848–5656, it replays that segment's forward pass, transiently holding up to 8 more activations (a peak of 8+8=168 + 8 = 16 units, a 4×4\times reduction) for a fixed ~33% increase in forward FLOPs. As LL grows the memory reduction factor grows like L/2\sqrt{L}/2, while the compute tax stays flat: exactly the trade that makes checkpointing more valuable the deeper the model gets.

Related concepts

  • GPUs for Deep Learning: mixed precision is the sibling memory-saving technique (halves activation size); checkpointing reduces activation count instead
  • MLOps Infrastructure & Orchestration: the resource-management trade-offs that make techniques like checkpointing part of a production training stack

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Gradient CheckpointingAutomatic Differentiation…MLOps Infrastructure & Or…GPUs for Deep Learning