Distributed Training at ScaleOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Distributed Training at Scale

Lesson 4 mapped a neural network onto a single GPU. Foundation models blow past what a single GPU (even a single node) can hold or compute in a reasonable time. This lesson covers the two walls that force training onto many GPUs at once, and the three families of parallelism that get around them.

Real-world examples: GPT- and Llama-scale pretraining runs on thousands of GPUs simultaneously; PyTorch's FSDP and Microsoft's DeepSpeed (ZeRO) are the memory-sharding libraries every large training run sits on top of; Megatron-LM's tensor-parallel matmul splitting; the reason LoRA/QLoRA (fine-tuning-alignment) are popular is that they let you dodge almost everything in this lesson by only training a small adapter.

Two walls, not one

  • The compute wall. A frontier model needs more FLOPs than one GPU can deliver in an acceptable wall-clock time, even running flat out.
  • The memory wall. A 70B-parameter model's weights, gradients, and optimizer state don't fit in one GPU's memory — regardless of how fast that GPU is.

These are different problems with different fixes. Compute wall → parallelize the work. Memory wall → shard the state. Confusing the two is the single most common conceptual mistake in this area.

Data parallelism: parallelize the work

Data parallelism (DP) replicates the full model on every GPU and shards the batch — each GPU runs forward/backward on its own slice of data, producing its own local gradients. Before the optimizer step, an all-reduce averages the gradients across all GPUs so every replica applies the identical update:

g=1G∑i=1Ggig = \frac{1}{G}\sum_{i=1}^{G} g_i

where GG is the number of GPUs and gig_i is GPU ii's local gradient. A ring all-reduce moves ≈2Ψ(G−1)/G\approx 2\Psi(G-1)/G worth of gradient data per GPU (Ψ\Psi = parameter count) — bandwidth cost that is nearly independent of GG for large GG, which is why DP scales well, provided the all-reduce overlaps with backward compute rather than stalling after it.

DP solves compute, not memory

Every GPU still holds a full copy of the parameters, gradients, and optimizer state. Doubling the GPU count doubles your throughput, but does nothing for a model whose optimizer state alone is too big for one GPU. That's the memory wall, and DP alone never touches it.

The memory wall, in bytes

For mixed-precision training with Adam, each parameter carries far more than its own 2 bytes. A standard accounting (fp16 weights + fp16 gradients + fp32 master weights + fp32 momentum + fp32 variance):

2Ψ⏟fp16 params+2Ψ⏟fp16 grads+4Ψ+4Ψ+4Ψ⏟fp32 optimizer state=16Ψ bytes\underbrace{2\Psi}_{\text{fp16 params}} + \underbrace{2\Psi}_{\text{fp16 grads}} + \underbrace{4\Psi + 4\Psi + 4\Psi}_{\text{fp32 optimizer state}} = 16\Psi \text{ bytes}

A 7B-parameter model needs 112 GB of model state under this accounting — before a single activation is stored. Naive DP puts all 16Ψ16\Psi bytes on every GPU, so adding GPUs never shrinks this number.

ZeRO / FSDP: shard the state instead of replicating it

ZeRO (Zero Redundancy Optimizer — the idea behind PyTorch's FSDP) keeps DP's batch-sharding but stops replicating state that doesn't need to be replicated. It shards the 16Ψ16\Psi bytes across the GG data-parallel GPUs in three increasing stages:

StageShardsPer-GPU memory
Baseline DPnothing16Ψ16\Psi
ZeRO-1 (PosP_{os})optimizer state4Ψ+12Ψ/G4\Psi + 12\Psi/G
ZeRO-2 (Pos+gP_{os+g})+ gradients2Ψ+14Ψ/G2\Psi + 14\Psi/G
ZeRO-3 (Pos+g+pP_{os+g+p})+ parameters16Ψ/G16\Psi/G

Each stage shards one more piece of state across the GG GPUs, gathering it back only for the instant it's needed (e.g. all-gathering a shard's parameters just before that shard's forward pass) and freeing it right after. ZeRO-3's 16Ψ/G16\Psi/G is the payoff: per-GPU memory now scales down with GPU count, exactly what naive DP couldn't do.

Sharding model state: ZeRO stages vs. GPU count

GPU 0GPU 1GPU 2GPU 3
fp16 params (2Ψ) fp16 grads (2Ψ) fp32 optimizer state (12Ψ)
bytes / param16.0Ψ
7B model, per GPU112.0 GB
fits on 80GB GPU?no

Per-GPU bytes-per-parameter (Ψ) for Baseline DP across 4 GPUs. Baseline data parallelism replicates all 16Ψ bytes on every GPU. Each ZeRO stage shards one more piece of state — optimizer state, then gradients, then parameters — across the group, shrinking the per-GPU total toward 16Ψ/G.

More sharding, more communication

Each ZeRO stage trades memory for communication: ZeRO-3 needs an extra all-gather of parameters on every forward and backward pass, on top of the gradient all-reduce DP already pays. This is why teams pick the lowest ZeRO stage that fits in memory rather than defaulting to ZeRO-3 — stage 1 or 2 is often enough and communicates less.

When sharding still isn't enough: model and pipeline parallelism

ZeRO shards state, but every GPU still needs to materialize the full activations of every layer during its forward pass. When a single layer is too large for that (or DP communication itself becomes the bottleneck), the model itself is split:

  • Tensor parallelism splits an individual matmul across GPUs — e.g. a weight matrix's columns live on different GPUs, each computing a partial GEMM (the same GEMM from Lesson 4, just partitioned), with a small all-reduce/all-gather to combine partial results. Communication is frequent and latency-sensitive, so tensor parallelism is normally kept within a fast-interconnect node.
  • Pipeline parallelism splits the model by layer: GPU 1 holds layers 1–8, GPU 2 holds layers 9–16, and so on, with activations passed forward between them like a pipeline. To keep every GPU busy, the batch is cut into MM micro-batches that flow through the stages back-to-back.

Pipeline parallelism's cost is the bubble: with PP pipeline stages, the first stage sits idle while later stages haven't started, and the last stage sits idle while earlier stages finish. The idle fraction is

bubble fraction=P−1P−1+M\text{bubble fraction} = \frac{P - 1}{P - 1 + M}

More micro-batches (MM) shrink the bubble — at the cost of smaller per-microbatch activations and more scheduling overhead.

Putting it together: 3D parallelism

Production training runs (Megatron-LM, DeepSpeed) combine all three axes at once — data parallelism across groups of GPUs, tensor parallelism within a node, pipeline parallelism across nodes — because each axis answers a different question:

SymptomAxis that helps
Training is too slowData parallelism (more GPUs on the batch)
Optimizer/gradients don't fitZeRO stage 1 or 2
Parameters themselves don't fitZeRO-3, or tensor parallelism
A single layer's activations don't fitTensor or pipeline parallelism

Exercise

You double the number of GPUs used for naive data parallelism (full model replicated on each). What happens to the per-GPU memory needed for parameters, gradients, and optimizer state?

Need a hint?

Data parallelism shards the batch, not the model state.

Exercise

For mixed-precision Adam training, total model state is 16Ψ bytes per parameter (2Ψ fp16 params + 2Ψ fp16 grads + 12Ψ fp32 optimizer state). Which ZeRO stage first shrinks per-GPU memory to scale down with the number of GPUs G, and what is it at full sharding?

Need a hint?

ZeRO-1 shards only the optimizer state; ZeRO-3 shards everything.

Exercise

A pipeline-parallel run splits the model across P = 5 stages and uses M = 15 micro-batches. Using bubble fraction = (P−1)/(P−1+M), what fraction of the time is spent in the pipeline bubble (idle)?

00.501
Need a hint?

(5−1) / (5−1+15) = 4/19.

Common mistakes

  1. Expecting data parallelism to solve a memory problem. DP replicates state; it never shards it. More GPUs under naive DP means more copies of the same too-big state, not a smaller one.
  2. Not overlapping the gradient all-reduce with backward compute. A DP implementation that waits for backward to finish before starting communication throws away most of DP's scalability.
  3. Reaching for tensor or pipeline parallelism before exhausting ZeRO. They solve a real problem (a layer or the full parameter set doesn't fit) but add real complexity and communication; ZeRO-1/2 alone is enough for most models that merely have a large optimizer footprint.
  4. Picking a micro-batch count without checking the bubble. Too few micro-batches leaves pipeline stages idle a large fraction of the time; the bubble fraction formula tells you directly how many you need.

Key takeaways

  • Two distinct walls: compute (not enough FLOPs) and memory (state doesn't fit), and they need different fixes.
  • Data parallelism replicates the model, shards the batch, and syncs gradients with all-reduce — it scales compute but not memory.
  • ZeRO/FSDP shards optimizer state, then gradients, then parameters across the data-parallel group, taking per-GPU memory from 16Ψ16\Psi down to 16Ψ/G16\Psi/G at full sharding (stage 3), at the cost of extra communication.
  • Tensor and pipeline parallelism split the model itself when sharding state still isn't enough; pipeline parallelism pays an idle "bubble" that shrinks with more micro-batches.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Distributed Training at ScaleNeural NetworksGPUs for Deep LearningMLOps Infrastructure & Or…PEFT: LoRA & QLoRA