Memory & Performance Optimization
A correct kernel and a fast kernel can differ by 10×. The gap is almost always memory: how threads touch global memory, how much data you reuse before discarding it, and how many warps you keep in flight. This lesson covers the four levers that decide real-world GPU performance.
Real-world examples: FlashAttention's whole point is tiling attention into shared memory to avoid materializing a huge matrix in slow global memory; cuBLAS GEMM is a master class in coalescing + tiling; "why is my custom kernel slow?" is, nine times out of ten, a coalescing or occupancy problem.
Memory coalescing
Global memory is delivered in wide transactions (e.g. 128-byte cache lines), not one float at a time. When the 32 threads of a warp access consecutive addresses, the hardware coalesces them into a single transaction — maximum bandwidth. When they access scattered or strided addresses, each thread may trigger its own transaction, wasting most of the bytes fetched.
Memory coalescing: stride decides bandwidth
Eight threads of a warp each read one memory cell. Global memory arrives in fixed transactions (here 4 cells each). Stride 1 packs the reads into the fewest transactions — coalesced, ~100% of fetched bytes used. Larger strides scatter reads across more transactions, so most fetched bytes are wasted and effective bandwidth collapses.
A stride-1 (coalesced) access is ~100% efficient; a stride- access can be as low as . This is why data layout matters: storing a batch of vectors row-major and having thread read column across rows is a coalescing disaster; transpose the layout so consecutive threads read consecutive addresses.
Coalescing is the #1 lever
Before tuning anything else, make sure consecutive threads read consecutive memory. A non-coalesced load pattern can cut effective bandwidth by 8–32×, dwarfing every other optimization. Often the fix is a layout change, not a code change.
Shared memory and tiling
Shared memory is a small (~48–228 KB/SM), fast, programmer-managed scratchpad shared by a block. The strategy: cooperatively load a tile of data from global memory into shared memory once, then have all the threads in the block reuse it many times.
The classic case is matrix multiply. Naively, computing re-reads each element of and from global memory times. Tiling loads a block of each into shared memory and reuses it, cutting global-memory traffic by a factor of :
__shared__ float Asub[T][T], Bsub[T][T]; // per-block scratchpad
// each thread loads one element of the tile, then:
__syncthreads(); // wait until the whole tile is loaded
for (int k = 0; k < T; ++k) // reuse the tile T times
acc += Asub[ty][k] * Bsub[k][tx];
__syncthreads(); // wait before overwriting the tile
The __syncthreads() barriers are essential: a thread must not read the tile before its neighbors have finished writing it.
Occupancy
Occupancy is the ratio of active warps on an SM to the hardware maximum. It matters because resident warps are what hide memory latency (Lesson 1) — when one stalls, the SM runs another.
Each SM has a fixed budget of registers and shared memory. A kernel that uses many registers per thread, or a lot of shared memory per block, lets fewer warps fit — lowering occupancy. The tension: tiling wants shared memory, but too much shared memory starves occupancy.
Higher occupancy isn't always faster
Occupancy is a means (latency hiding), not an end. Past the point where latency is fully hidden, more occupancy does nothing, and a low-occupancy kernel with heavy per-thread reuse (lots of registers) can beat a high-occupancy one. Measure; don't maximize occupancy blindly.
The roofline model
The roofline model answers the key diagnostic question: is my kernel limited by compute or by memory? Define arithmetic intensity as FLOPs performed per byte of memory moved:
Achievable performance is capped by the lower of two ceilings:
- Memory-bound (low , left of the ridge): performance , limited by bandwidth. Element-wise ops, vector add, and activations live here.
- Compute-bound (high , right of the ridge): performance , limited by arithmetic throughput. Large matrix multiplies live here.
The ridge point is the intensity where the two ceilings meet. Optimizing a memory-bound kernel means raising its intensity (fuse ops, reuse data via tiling) or improving its bandwidth (coalesce); throwing more FLOPs at it does nothing.
Putting it together
A fast kernel typically: (1) coalesces global accesses, (2) tiles into shared memory to raise arithmetic intensity and cut global traffic, (3) keeps enough warps resident to hide whatever latency remains. The roofline tells you which of these to chase first.
Exercise
Why is it important that the 32 threads of a warp read consecutive global-memory addresses?
Need a hint?
Global memory is delivered in wide transactions, not single floats.
Exercise
In a tiled matrix multiply, threads cooperatively load a T×T tile into shared memory before computing. What does this primarily achieve?
Need a hint?
How many times is each loaded value reused before being discarded?
Exercise
A kernel has very low arithmetic intensity (few FLOPs per byte moved), e.g. an element-wise activation. The roofline model classifies it as memory-bound. What will speed it up?
Need a hint?
Adding FLOPs to something limited by bandwidth does nothing.
Common mistakes
- Strided/scattered global access: the single biggest bandwidth killer; fix the layout so consecutive threads hit consecutive addresses.
- Skipping
__syncthreads()around shared-memory tiles: a race where threads read a tile before it's fully written. - Maximizing occupancy for its own sake: it only helps until latency is hidden; per-thread reuse can be worth lower occupancy.
- Optimizing compute on a memory-bound kernel: adding FLOPs to something capped by bandwidth changes nothing; raise arithmetic intensity instead.
Key takeaways
- Coalescing: consecutive threads must read consecutive memory; non-coalesced access can waste 8–32× of bandwidth.
- Tiling into shared memory reuses data and cuts global traffic by the tile factor (the heart of fast GEMM and FlashAttention).
- Occupancy = resident warps ÷ max; it exists to hide latency, and more isn't always better.
- The roofline model classifies a kernel as memory-bound or compute-bound via arithmetic intensity , telling you what to optimize.
Related concepts
- GPU Architecture & the Execution Model: the memory hierarchy and latency hiding these techniques exploit
- GPUs for Deep Learning: applying coalescing, tiling, and the roofline to real ML kernels
- Deployment Patterns & Model Compression: when memory and bandwidth limits drive compression choices