Distributed Training at Scale
Lesson 4 mapped a neural network onto a single GPU. Foundation models blow past what a single GPU (even a single node) can hold or compute in a reasonable time. This lesson covers the two walls that force training onto many GPUs at once, and the three families of parallelism that get around them.
Real-world examples: GPT- and Llama-scale pretraining runs on thousands of GPUs simultaneously; PyTorch's FSDP and Microsoft's DeepSpeed (ZeRO) are the memory-sharding libraries every large training run sits on top of; Megatron-LM's tensor-parallel matmul splitting; the reason LoRA/QLoRA (fine-tuning-alignment) are popular is that they let you dodge almost everything in this lesson by only training a small adapter.
Two walls, not one
- The compute wall. A frontier model needs more FLOPs than one GPU can deliver in an acceptable wall-clock time, even running flat out.
- The memory wall. A 70B-parameter model's weights, gradients, and optimizer state don't fit in one GPU's memory — regardless of how fast that GPU is.
These are different problems with different fixes. Compute wall → parallelize the work. Memory wall → shard the state. Confusing the two is the single most common conceptual mistake in this area.
Data parallelism: parallelize the work
Data parallelism (DP) replicates the full model on every GPU and shards the batch — each GPU runs forward/backward on its own slice of data, producing its own local gradients. Before the optimizer step, an all-reduce averages the gradients across all GPUs so every replica applies the identical update:
where is the number of GPUs and is GPU 's local gradient. A ring all-reduce moves worth of gradient data per GPU ( = parameter count) — bandwidth cost that is nearly independent of for large , which is why DP scales well, provided the all-reduce overlaps with backward compute rather than stalling after it.
DP solves compute, not memory
Every GPU still holds a full copy of the parameters, gradients, and optimizer state. Doubling the GPU count doubles your throughput, but does nothing for a model whose optimizer state alone is too big for one GPU. That's the memory wall, and DP alone never touches it.
The memory wall, in bytes
For mixed-precision training with Adam, each parameter carries far more than its own 2 bytes. A standard accounting (fp16 weights + fp16 gradients + fp32 master weights + fp32 momentum + fp32 variance):
A 7B-parameter model needs 112 GB of model state under this accounting — before a single activation is stored. Naive DP puts all bytes on every GPU, so adding GPUs never shrinks this number.
ZeRO / FSDP: shard the state instead of replicating it
ZeRO (Zero Redundancy Optimizer — the idea behind PyTorch's FSDP) keeps DP's batch-sharding but stops replicating state that doesn't need to be replicated. It shards the bytes across the data-parallel GPUs in three increasing stages:
| Stage | Shards | Per-GPU memory |
|---|---|---|
| Baseline DP | nothing | |
| ZeRO-1 () | optimizer state | |
| ZeRO-2 () | + gradients | |
| ZeRO-3 () | + parameters |
Each stage shards one more piece of state across the GPUs, gathering it back only for the instant it's needed (e.g. all-gathering a shard's parameters just before that shard's forward pass) and freeing it right after. ZeRO-3's is the payoff: per-GPU memory now scales down with GPU count, exactly what naive DP couldn't do.
Sharding model state: ZeRO stages vs. GPU count
Per-GPU bytes-per-parameter (Ψ) for Baseline DP across 4 GPUs. Baseline data parallelism replicates all 16Ψ bytes on every GPU. Each ZeRO stage shards one more piece of state — optimizer state, then gradients, then parameters — across the group, shrinking the per-GPU total toward 16Ψ/G.
More sharding, more communication
Each ZeRO stage trades memory for communication: ZeRO-3 needs an extra all-gather of parameters on every forward and backward pass, on top of the gradient all-reduce DP already pays. This is why teams pick the lowest ZeRO stage that fits in memory rather than defaulting to ZeRO-3 — stage 1 or 2 is often enough and communicates less.
When sharding still isn't enough: model and pipeline parallelism
ZeRO shards state, but every GPU still needs to materialize the full activations of every layer during its forward pass. When a single layer is too large for that (or DP communication itself becomes the bottleneck), the model itself is split:
- Tensor parallelism splits an individual matmul across GPUs — e.g. a weight matrix's columns live on different GPUs, each computing a partial GEMM (the same GEMM from Lesson 4, just partitioned), with a small all-reduce/all-gather to combine partial results. Communication is frequent and latency-sensitive, so tensor parallelism is normally kept within a fast-interconnect node.
- Pipeline parallelism splits the model by layer: GPU 1 holds layers 1–8, GPU 2 holds layers 9–16, and so on, with activations passed forward between them like a pipeline. To keep every GPU busy, the batch is cut into micro-batches that flow through the stages back-to-back.
Pipeline parallelism's cost is the bubble: with pipeline stages, the first stage sits idle while later stages haven't started, and the last stage sits idle while earlier stages finish. The idle fraction is
More micro-batches () shrink the bubble — at the cost of smaller per-microbatch activations and more scheduling overhead.
Putting it together: 3D parallelism
Production training runs (Megatron-LM, DeepSpeed) combine all three axes at once — data parallelism across groups of GPUs, tensor parallelism within a node, pipeline parallelism across nodes — because each axis answers a different question:
| Symptom | Axis that helps |
|---|---|
| Training is too slow | Data parallelism (more GPUs on the batch) |
| Optimizer/gradients don't fit | ZeRO stage 1 or 2 |
| Parameters themselves don't fit | ZeRO-3, or tensor parallelism |
| A single layer's activations don't fit | Tensor or pipeline parallelism |
Exercise
You double the number of GPUs used for naive data parallelism (full model replicated on each). What happens to the per-GPU memory needed for parameters, gradients, and optimizer state?
Need a hint?
Data parallelism shards the batch, not the model state.
Exercise
For mixed-precision Adam training, total model state is 16Ψ bytes per parameter (2Ψ fp16 params + 2Ψ fp16 grads + 12Ψ fp32 optimizer state). Which ZeRO stage first shrinks per-GPU memory to scale down with the number of GPUs G, and what is it at full sharding?
Need a hint?
ZeRO-1 shards only the optimizer state; ZeRO-3 shards everything.
Exercise
A pipeline-parallel run splits the model across P = 5 stages and uses M = 15 micro-batches. Using bubble fraction = (P−1)/(P−1+M), what fraction of the time is spent in the pipeline bubble (idle)?
Need a hint?
(5−1) / (5−1+15) = 4/19.
Common mistakes
- Expecting data parallelism to solve a memory problem. DP replicates state; it never shards it. More GPUs under naive DP means more copies of the same too-big state, not a smaller one.
- Not overlapping the gradient all-reduce with backward compute. A DP implementation that waits for backward to finish before starting communication throws away most of DP's scalability.
- Reaching for tensor or pipeline parallelism before exhausting ZeRO. They solve a real problem (a layer or the full parameter set doesn't fit) but add real complexity and communication; ZeRO-1/2 alone is enough for most models that merely have a large optimizer footprint.
- Picking a micro-batch count without checking the bubble. Too few micro-batches leaves pipeline stages idle a large fraction of the time; the bubble fraction formula tells you directly how many you need.
Key takeaways
- Two distinct walls: compute (not enough FLOPs) and memory (state doesn't fit), and they need different fixes.
- Data parallelism replicates the model, shards the batch, and syncs gradients with all-reduce — it scales compute but not memory.
- ZeRO/FSDP shards optimizer state, then gradients, then parameters across the data-parallel group, taking per-GPU memory from down to at full sharding (stage 3), at the cost of extra communication.
- Tensor and pipeline parallelism split the model itself when sharding state still isn't enough; pipeline parallelism pays an idle "bubble" that shrinks with more micro-batches.
Related concepts
- GPUs for Deep Learning: the GEMM that tensor parallelism partitions, and the memory/compute-bound framing this lesson builds on
- MLOps Infrastructure & Orchestration: where distributed-training job schedulers (Ray, Kubernetes) fit into the wider platform
- PEFT / LoRA / QLoRA: why training a small adapter instead of the full model sidesteps most of this lesson's memory wall