Modern Attention Mechanisms
The original Transformer paper introduced an architecture that worked. The models in production today (LLaMA, Mistral, GPT-4, Claude) are built on that same architecture but with a set of targeted engineering improvements. Each one solves a specific bottleneck: cross-attention extends the design to translation and summarization; MQA/GQA shrink the memory cost of long-context inference; RoPE gives relative position awareness for free; FlashAttention eliminates the quadratic memory wall; and sparse patterns let attention scale to hundreds of thousands of tokens.
Encoder–decoder cross-attention
The classic Transformer is an encoder–decoder model: an encoder reads the source sequence, and a decoder generates the target sequence one token at a time. Connecting them requires a mechanism for the decoder to "look at" the encoded source — that is cross-attention.
How the queries and keys/values split
In cross-attention, the queries () come from the decoder and the keys and values (, ) come from the encoder's output:
The result is: each decoder position computes a weighted blend of encoder positions. The attention score measures how relevant encoder position is when generating decoder position .
Masking strategy
| Stack | Mask | Why |
|---|---|---|
| Encoder self-attention | None: bidirectional | Needs full context to understand the source |
| Decoder self-attention | Causal (lower-triangular) | Autoregressive: can't peek at future targets |
| Decoder cross-attention | None (over encoder) | All encoder positions are always available |
Worked example: translating "Hola" → "Hello"
Suppose the encoder produces representations for ["Hola", <EOS>] (2 tokens), and the decoder has generated [<BOS>] and is predicting "Hello".
- are fixed: they are the encoder output matrix, shape .
- The decoder's query for position 0 has shape .
- The attention score is , shape .
- After softmax, e.g. — the decoder attends heavily to "Hola".
- The context vector is .
- This context is passed to the FFN to produce the hidden state that is projected to predict "Hello".
Cross-attention is the encoder–decoder bridge
In encoder–decoder models every decoder layer has three sub-layers: (1) causal self-attention over what has been generated so far, (2) cross-attention over the encoder output, (3) the feed-forward network. If you remove the encoder and the cross-attention you get a decoder-only model (GPT style).
Multi-Head Attention variants
The KV cache problem
During autoregressive inference, the model generates one token at a time. To avoid recomputing keys and values for all previous tokens at each step, frameworks maintain a KV cache: a buffer storing and for every past position, for every layer.
KV cache size per layer:
where is the current sequence length and is the number of heads. For a LLaMA-2-70B-scale model with full MHA: heads, , tokens, bfloat16 (2 bytes) → bytes ≈ 134 MB per layer, times 80 layers ≈ 10.7 GB just for the KV cache at max context — per sequence being served. Multi-query and grouped-query attention attack this directly.
Multi-Query Attention (MQA)
In MQA there is a single shared and head across all query heads. Each query head still has its own , but all heads share one and one :
KV cache shrinks by a factor of : instead of storing copies of and , you store one. Quality drops slightly; true MQA is rare in practice — most modern LLMs prefer GQA instead.
Grouped-Query Attention (GQA)
GQA generalizes MQA: split the query heads into groups (). Each group shares one and one head. is MQA; is standard MHA.
Worked example: 8 heads, GQA with G = 2
- 8 Q-heads (independent )
- 2 K-heads, 2 V-heads (shared within each group of 4)
- KV cache = — 4× smaller than MHA
| Variant | Q heads | K/V heads | KV cache factor | Notes |
|---|---|---|---|---|
| MHA | Original | |||
| GQA | smaller | Mistral, LLaMA-2/3 | ||
| MQA | smaller | Extreme; PaLM, Falcon |
GQA in practice
Mistral 7B uses GQA with query heads and KV heads — a 4× KV cache reduction. LLaMA-2 70B and LLaMA-3 70B both use GQA ( query heads, KV heads) — an 8× cache reduction. Quality comparisons show GQA with to is nearly indistinguishable from full MHA.
Rotary Positional Encoding (RoPE)
The idea
Instead of adding a sinusoidal vector to the token embedding (as the original Transformer does), RoPE encodes position by rotating the query and key vectors before computing dot products. No extra embedding is needed; position is injected directly into the attention score.
The rotation
For a query vector , group pairs of dimensions and rotate them by angle where:
- is the absolute position of the token in the sequence
- , with (following the sinusoidal PE convention)
The same rotation is applied to at position .
Key property: relative distance
Rotating by angle and by angle , the dot product of each pair picks up — the rotation addition identity. The absolute positions and cancel; only the offset remains. The same applies across all dimension pairs simultaneously.
When you compute the dot product of a rotated query at position with a rotated key at position :
where is a rotation matrix depending only on the relative offset . This means attention scores are automatically sensitive to how far apart tokens are, not where they are in absolute terms — a property that improves generalization to sequence lengths not seen during training.
import numpy as np
def rope_rotate_pair(q2i, q2i1, m, theta_i):
"""Rotate a pair of query dimensions (q_{2i}, q_{2i+1}) at position m."""
cos_t = np.cos(m * theta_i)
sin_t = np.sin(m * theta_i)
return (cos_t * q2i - sin_t * q2i1,
sin_t * q2i + cos_t * q2i1)
d, base = 8, 10000
thetas = [base ** (-2 * i / d) for i in range(d // 2)] # θ_0 … θ_{d/2-1}
q = np.array([1.0, 0.5, -0.3, 0.8, 0.1, -0.2, 0.6, 0.4])
m = 3 # token at position 3
q_rot = np.zeros_like(q)
for i, theta_i in enumerate(thetas):
q_rot[2*i], q_rot[2*i+1] = rope_rotate_pair(q[2*i], q[2*i+1], m, theta_i)
print("original q:", q)
print("rotated q:", q_rot.round(4))
RoPE is used everywhere modern
RoPE was introduced by Su et al. (2021) and has since been adopted by LLaMA (all versions), Mistral, GPT-NeoX, Falcon, and Qwen. Its relative-distance property also makes it easier to extend context beyond the training length via techniques such as YaRN and NTK-aware scaling.
FlashAttention
The memory wall in standard attention
Standard attention:
requires materializing the attention score matrix in full. For tokens and float16, that is MB per head per layer — stored in HBM (the GPU's slow main memory). With multiple heads and layers, this becomes prohibitive and requires many slow round-trips between HBM and the GPU's fast on-chip SRAM.
Tiled computation
FlashAttention (Dao et al., 2022) reorders the computation to never store the full matrix:
- Split , , and into blocks that fit in SRAM (fast on-chip memory).
- For each block of , iterate over all blocks of and , computing partial softmax values and accumulating the weighted sum of on-chip.
- Use the online softmax trick to maintain a running max and normalizer so the partial results are numerically stable without seeing all scores first.
Result: the full matrix is never written to HBM. Memory scales as instead of . The math is exactly equivalent to standard attention — no approximation.
| Metric | Standard attention | FlashAttention |
|---|---|---|
| HBM memory | ||
| Approximation | Exact | Exact |
| Speedup | 1× | 2–4× (practical) |
| Key idea | — | Tile + online softmax |
Why fewer HBM reads wins
On modern GPUs, compute (FLOPs) is cheap; bandwidth (reading/writing HBM) is expensive. Standard attention is memory-bandwidth bound, not compute bound. FlashAttention reads each element of , , only once per pass, replacing many slow HBM accesses with fast SRAM operations — that is where the 2–4× wall-clock speedup comes from.
Watch the online softmax run
FlashAttention — streaming a softmax you cannot see all of
Standard attention materialises the whole N×N score matrix before it can take a softmax, because a softmax needs its denominator — which needs every score. At N = 8192 in float16 that is 128 MB **per head per layer**, and it scales quadratically: N = 131,072 needs 32.0 GB for a single head. The whole of FlashAttention follows from refusing to build that matrix, so the question is how to normalise a softmax you have not finished seeing.
| N | N² scores | standard | tiled (this trace's blocks) |
|---|---|---|---|
| 1,024 | 1.0e+6 | 2 MB | 24 KB |
| 4,096 | 1.7e+7 | 32 MB | 96 KB |
| 8,192 | 6.7e+7 | 128 MB | 192 KB |
| 32,768 | 1.1e+9 | 2.0 GB | 768 KB |
| 131,072 | 1.7e+10 | 32.0 GB | 3072 KB |
The tiled column is not a smaller constant on the same curve — it is a different curve. Scores live only inside one B_r × B_c tile; everything else that grows with N is Q, K, V themselves, which are O(N·d) and had to exist anyway.
A 12-key attention row computed twice: once the standard way, materialising every score, and once in three tiles that never hold more than four. Watch the rescale factor in block 1, where a larger score arrives, the running maximum moves, and one scalar exp(m_old − m_new) corrects every partial result accumulated so far — that single factor is what makes a streamed softmax possible. Then two measured payoffs: the page's claim of exact equivalence holds to the last bit of float64 across all 48 output components, and deleting the running max fails not gradually but completely, at exactly ln(MAX_FLOAT64) = 709.78.
What to notice:
- Frame 4: the rescale is the whole trick. Block 1 contains a score larger than anything seen so far, so the running maximum moves from 0.634 to 0.862. Everything accumulated from block 0 was exponentiated against the old maximum and is now wrong by exactly . One scalar repairs an arbitrary amount of history — that is what lets a softmax be computed without ever seeing all its inputs.
- Frame 5: a rescale of exactly 1. The last block's maximum is below the running one, so nothing needs correcting. The rescale is not a per-block cost; it fires only when the maximum actually moves.
- Frame 6: "exactly equivalent" is testable, and it passes. Across all 48 output components the largest difference from standard attention is , i.e. float64 rounding. FlashAttention is not a fast approximation like sparse or low-rank attention — it computes the same function by a different association of the same arithmetic. Any output difference is a bug, never a tuning knob.
- Frame 7: why step 3 says "numerically stable". Delete the running max and the method is algebraically identical and practically dead: finite at a max score of 704.1, NaN at 751.0. The threshold is not empirical — falls exactly between those rows. Above it returns Inf, and Inf/Inf is NaN.
That last failure is worth remembering because it hides well: trained networks rarely produce scores near 709, so the bug survives review and then surfaces on a long context, an untrained model, or a lower-precision format — as NaN rather than as a slightly wrong number.
FlashAttention-2 (2023) and FlashAttention-3 (2024) push further with better work partitioning across thread blocks and tensor-core utilization, but the core idea (tile and keep data in SRAM) is unchanged.
Sliding-window and sparse attention
The quadratic wall
Full self-attention is in both time and memory. For tokens (Claude's context), that is billion attention scores per layer — even with FlashAttention's memory, the computation is still FLOPs.
Sliding-window attention
Each token attends only to the nearest tokens (a window of size centered at itself, one-sided for causal models):
Complexity: instead of . With and this is a 31× reduction.
Local + global tokens
Pure sliding-window loses long-range coherence — position 0 cannot directly influence position . Two common fixes:
- Longformer / BigBird: a handful of designated "global" tokens (e.g.,
[CLS]) attend to and receive attention from all positions, carrying long-range information. - Mistral 7B: uses sliding-window attention in lower layers (cheap local features) and full attention in upper layers (long-range integration).
Local vs global coherence trade-off
Sliding-window limits each token's receptive field to tokens. Information from far-away tokens reaches position only by propagating through many layers — the same limitation RNNs had. For tasks requiring tight long-range dependencies (complex reasoning, document summarization) pure local attention may degrade quality; combining local + global tokens mitigates this.
RMSNorm and SwiGLU
Modern LLMs swap out two components of the original Transformer: LayerNorm becomes RMSNorm and the ReLU FFN becomes SwiGLU. Both are small but consistent wins.
RMSNorm
Standard LayerNorm centers and scales:
RMSNorm drops the mean-centering step entirely, normalizing only by the root mean square:
No parameter (the mean shift is dropped). Fewer operations, no mean subtraction, empirically equivalent quality. Used by LLaMA, T5, PaLM.
SwiGLU
The original FFN uses two weight matrices and ReLU:
SwiGLU replaces this with a gated structure using three weight matrices:
where (also called Swish). The gating term acts as a smooth, learned filter on the features produced by .
import numpy as np
def silu(x):
return x / (1 + np.exp(-x)) # x * sigmoid(x)
def swiglu(x, W1, W3, W2):
gate = silu(x @ W3) # gating signal
hidden = (x @ W1) * gate # element-wise gating
return hidden @ W2 # project back
# Tiny example: d_model=4, d_ff=8 (usually 8/3 * d_model to match param count)
rng = np.random.RandomState(0)
d_model, d_ff = 4, 8
W1 = rng.randn(d_model, d_ff) * 0.1
W3 = rng.randn(d_model, d_ff) * 0.1
W2 = rng.randn(d_ff, d_model) * 0.1
x = rng.randn(3, d_model) # 3 tokens
out = swiglu(x, W1, W3, W2)
print("SwiGLU output shape:", out.shape) # (3, 4) — same as input
SwiGLU's extra weight matrix
SwiGLU adds a third projection matrix compared to the standard FFN. To keep total parameter count the same, the hidden dimension is reduced: instead of , SwiGLU uses (rounded up to a multiple of 64 for hardware efficiency). LLaMA uses this ratio throughout.
Exercise
In encoder-decoder cross-attention (e.g., machine translation), where do the Queries, Keys, and Values come from?
Need a hint?
The decoder wants to 'look up' information from the encoder's output.
Exercise
A model with 16 Q-heads uses Grouped-Query Attention (GQA) with 4 KV groups. Compared to Multi-Head Attention (MHA), how large is the KV cache?
Need a hint?
KV cache scales with the number of K/V heads, not Q heads.
Exercise
What is the key advantage of Rotary Positional Encoding (RoPE) over additive sinusoidal encodings?
Need a hint?
Think about how RoPE encodes position and what property the dot product has.
Common mistakes
- Cross-attention Q/K/V source confusion: Q comes from the decoder; K and V come from the encoder. Reversing this makes the decoder unable to query the source.
- Forgetting the causal mask in decoder self-attention: the decoder's self-attention still needs a causal mask; only the cross-attention step attends to the full encoder output without a causal constraint.
- Expecting MQA/GQA to improve quality: these are inference optimizations. They reduce KV cache size at a small quality cost; they don't improve perplexity and shouldn't be chosen for quality reasons.
- Thinking FlashAttention changes the result: FlashAttention is numerically identical to standard attention. If outputs differ, it's a precision or implementation bug, not an intentional trade-off.
- Applying RoPE only to Q: RoPE must be applied to both and for the relative-distance property to hold. Rotating only Q breaks the math.
- Confusing SwiGLU's : the SwiGLU FFN uses (not ) to match parameter counts with the two-matrix ReLU FFN.
Key takeaways
- Cross-attention bridges encoder and decoder: Q from decoder, K/V from encoder; no causal mask on the cross-attention step.
- MQA/GQA reduce KV cache memory at inference by sharing K/V heads across query head groups — GQA with is the current sweet spot.
- RoPE injects relative position into attention scores by rotating Q and K vectors; the dot product then depends only on position difference, not absolute position.
- FlashAttention tiles the attention computation to avoid materializing the matrix, achieving exact attention with memory and 2–4× wall-clock speedup.
- Sliding-window attention reduces attention complexity from to for long contexts; global tokens preserve long-range connectivity.
- RMSNorm drops mean-centering from LayerNorm; SwiGLU adds learned gating to the FFN — both are small, consistent quality improvements used in all major modern LLMs.
Related concepts
- Multi-Head Attention & Positional Encoding: the MHA foundation that MQA/GQA extend
- The Transformer Architecture: residuals, LayerNorm, encoder/decoder families
- Self-Attention: the core QKV mechanism
- Attention Mechanisms: the full taxonomy: self/cross/causal, MHA/MQA/GQA, and FlashAttention
- Positional Encodings: sinusoidal, RoPE, ALiBi, and their length-extrapolation tradeoffs