GPUs for Deep Learning
Everything in the first three lessons exists to make one thing fast: the arithmetic of neural networks. This lesson connects the hardware to the workload — why training is dominated by matrix multiplication, the specialized units that accelerate it, and the techniques (mixed precision, fusion) that frameworks use to keep the GPU's arithmetic units fed.
Real-world examples: the matmuls inside every linear layer and attention block; tensor cores doing the heavy lifting in mixed-precision training; torch.compile and FlashAttention fusing kernels to dodge memory traffic; batching at inference time to push a memory-bound decode into a more efficient regime.
Why matrix multiply dominates
A forward pass through a linear layer is — a matrix multiply. Attention is a stack of matrix multiplies. Convolutions are lowered to matrix multiplies (im2col) or implemented as specialized GEMMs. Across a real model, the overwhelming majority of FLOPs are matrix multiplies (GEMM — General Matrix Multiply).
This is a gift, because GEMM is the ideal GPU workload from Lesson 1: massively parallel (every output element is independent), high arithmetic intensity (each loaded value is reused times), and a regular, coalescable access pattern. GPU vendors therefore pour enormous effort into one routine, and you inherit it through cuBLAS/cuDNN.
For large the FLOPs () grow faster than the bytes (), so arithmetic intensity rises — GEMM is firmly compute-bound, exactly where the GPU is strongest.
Tensor cores
Modern GPUs add tensor cores: units that compute a small matrix multiply-accumulate ( on, say, 16×16 tiles) in a single operation, rather than one scalar multiply-add per ALU. They deliver many times the throughput of the general-purpose cores — for matmul-shaped work in supported precisions.
Tensor cores want the right shapes and types
Tensor cores engage for GEMM-like operations in reduced precision (FP16/BF16/TF32/FP8). To use them well, matrix dimensions should be multiples of the tile size (e.g. 8 or 16) and data should be in a supported low-precision format. A matmul with awkward dimensions or in FP32 may fall back to the slower general cores, which is why padding to "nice" shapes can speed a model up.
Mixed precision
Training in pure FP32 wastes the tensor cores and doubles memory traffic. Mixed-precision training keeps a master copy of the weights in FP32 but does the matmuls in 16-bit (FP16 or BF16):
- Half the memory and bandwidth for activations and gradients, and bandwidth is often the limit.
- Tensor cores engage, multiplying throughput.
- Numerics are managed: FP16 has a narrow range, so frameworks use loss scaling to keep small gradients from underflowing to zero; BF16 has FP32's exponent range and usually needs no scaling.
The accumulation is still done in FP32 inside the tensor core, so accuracy is preserved while the expensive multiplies run in 16-bit.
Mixed precision saves memory by shrinking each activation's size. A sibling technique, gradient checkpointing, instead saves memory by shrinking the number of activations kept around — only a -spaced subset is cached, and the rest are recomputed during backprop.
Kernel fusion
Each kernel launch reads its inputs from global memory and writes its outputs back. A chain of small element-wise ops (bias → activation → dropout) pays that round-trip every step, even though each op does almost no arithmetic. These ops are memory-bound (low arithmetic intensity from Lesson 3).
Kernel fusion merges them into one kernel: load once, do all the arithmetic in registers, write once.
For memory-bound chains this is a near- speedup. This is exactly what torch.compile, XLA, and TensorRT do automatically, and what FlashAttention does by hand — fusing the attention softmax-and-matmul so the giant score matrix never touches global memory.
Compute-bound vs memory-bound: the practical divide
| Operation | Bound | Why | Lever |
|---|---|---|---|
| Large matmul / conv | compute | high arithmetic intensity | tensor cores, low precision |
| Element-wise (activation, bias, norm) | memory | ~1 FLOP/byte | fusion |
| Attention (long sequence) | memory (naive) | materializes scores | fusion + tiling (FlashAttention) |
| LLM decode (batch 1) | memory | re-reads all weights per token | batching, KV-cache |
Inference is often memory-bound
Generating one token at a time with batch size 1 re-reads the entire weight matrix to produce a single column of output — tiny arithmetic intensity, so it's bandwidth-limited and leaves the compute units idle. Batching many requests reuses each loaded weight across many tokens, pushing the operation toward compute-bound and dramatically raising throughput. This is the core reason inference servers batch.
Putting that table on one diagram
Which limit is the one you are hitting
Attainable throughput is min(peak FLOP/s, bandwidth × arithmetic intensity), both axes log. The sloped part is the memory roof, the flat part the compute roof, and the corner between them is the ridge point at peak ÷ bandwidth. A kernel left of the ridge is bandwidth-bound and no amount of faster arithmetic helps it. Hardware numbers are vendor peak specs.
Every row of the table above is a point on this plot, and where it sits relative to the ridge point (peak FLOP/s ÷ bandwidth) is the whole diagnosis. Left of the ridge you are waiting on memory and faster arithmetic buys nothing; right of it you are actually using the machine.
Toggle FP32 to tensor cores and watch the ridge move. On an A100 it goes from FLOP/byte to — 16× further to the right. This is the part that trips people up: making the arithmetic 16× faster did not make more kernels compute-bound, it made more of them memory-bound. GEMM n=256 at 85 FLOP/byte is comfortably compute-bound in FP32 and firmly bandwidth-bound in FP16. Every generation of faster tensor cores raises the intensity bar your kernels have to clear.
Then drag the batch slider, because for LLM decode the arithmetic intensity is the batch size. Reading an weight matrix costs bytes in FP16 and does FLOPs for a batch of , so
That converts "batch your inference server" from advice into a target:
| batch | attainable on A100 tensor cores | % of peak |
|---|---|---|
| 1 | 1.6 TFLOP/s | 0.5% |
| 8 | 12.4 TFLOP/s | 4.0% |
| 64 | 99.5 TFLOP/s | 31.9% |
| 201 | 312 TFLOP/s | 100% |
Batch 1 decode runs a TFLOP/s machine at 1.6. You need on an A100 ( on an H100) before the weights you dragged out of HBM are being reused enough to leave the memory roof. That single number is why continuous batching exists.
Note what the fusion rows show, too: fusing three element-wise ops moves them from to FLOP/byte — a genuine 3× less traffic, and still nowhere near the ridge. Fusion does not make a memory-bound kernel compute-bound; it just makes you do the memory-bound thing fewer times.
Exercise
Multiplying two square N×N matrices with N = 100 costs 2·N³ FLOPs. How many million FLOPs is that?
Need a hint?
2 × 100³ = 2 × 1,000,000.
Exercise
Why does mixed-precision training (FP16/BF16 matmuls with FP32 accumulation) speed up deep learning?
Need a hint?
Think bandwidth and specialized hardware units.
Exercise
A chain of k small element-wise ops (bias → activation → dropout) is memory-bound. Fusing them into one kernel changes the number of global-memory round-trips from 2k to what, and why does it help?
Need a hint?
Load once, compute everything in registers, write once.
Common mistakes
- Training in FP32 by default: leaves tensor cores and half your bandwidth on the table; use BF16/FP16 mixed precision.
- Ignoring kernel fusion for element-wise chains: paying memory round-trips for ops that should be one; let the compiler fuse, or fuse by hand.
- Awkward matmul dimensions: shapes that aren't multiples of the tile size can miss tensor cores entirely.
- Serving at batch size 1 and blaming the GPU: decode is memory-bound; batch to convert idle bandwidth waste into throughput.
Key takeaways
- Matrix multiply (GEMM) dominates deep-learning FLOPs and is the ideal compute-bound GPU workload.
- Tensor cores accelerate matmul-shaped work in reduced precision; feed them nice shapes and low-precision data.
- Mixed precision (FP16/BF16 matmuls, FP32 accumulation, loss scaling) halves bandwidth and engages tensor cores with managed numerics.
- Kernel fusion collapses memory-bound op chains from to global round-trips; batching rescues memory-bound inference by reusing loaded weights.
Related concepts
- Memory & Performance Optimization: the roofline and tiling ideas this lesson applies to ML kernels
- Layers and the Forward Pass: the matrix multiplies at the heart of every network
- Inference Optimization & Serving: batching, KV-cache, and serving the memory-bound decode