Modern Attention MechanismsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Modern Attention Mechanisms

The original Transformer paper introduced an architecture that worked. The models in production today (LLaMA, Mistral, GPT-4, Claude) are built on that same architecture but with a set of targeted engineering improvements. Each one solves a specific bottleneck: cross-attention extends the design to translation and summarization; MQA/GQA shrink the memory cost of long-context inference; RoPE gives relative position awareness for free; FlashAttention eliminates the quadratic memory wall; and sparse patterns let attention scale to hundreds of thousands of tokens.

Encoder–decoder cross-attention

The classic Transformer is an encoder–decoder model: an encoder reads the source sequence, and a decoder generates the target sequence one token at a time. Connecting them requires a mechanism for the decoder to "look at" the encoded source — that is cross-attention.

How the queries and keys/values split

In cross-attention, the queries (QQ) come from the decoder and the keys and values (KK, VV) come from the encoder's output:

Q=Xdec WQ,K=Xenc WK,V=Xenc WVQ = X_{\text{dec}}\,W_Q, \qquad K = X_{\text{enc}}\,W_K, \qquad V = X_{\text{enc}}\,W_V CrossAttn(Xdec, Xenc)=softmax ⁣(QK⊤dk)V\text{CrossAttn}(X_{\text{dec}},\,X_{\text{enc}}) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

The result is: each decoder position computes a weighted blend of encoder positions. The attention score Qi⋅KjQ_i \cdot K_j measures how relevant encoder position jj is when generating decoder position ii.

Masking strategy

StackMaskWhy
Encoder self-attentionNone: bidirectionalNeeds full context to understand the source
Decoder self-attentionCausal (lower-triangular)Autoregressive: can't peek at future targets
Decoder cross-attentionNone (over encoder)All encoder positions are always available

Worked example: translating "Hola" → "Hello"

Suppose the encoder produces representations for ["Hola", <EOS>] (2 tokens), and the decoder has generated [<BOS>] and is predicting "Hello".

  • K,VK, V are fixed: they are the encoder output matrix, shape (2,dk)(2, d_k).
  • The decoder's query for position 0 has shape (1,dk)(1, d_k).
  • The attention score is s=Q⋅K⊤/dks = Q \cdot K^\top / \sqrt{d_k}, shape (1,2)(1, 2).
  • After softmax, e.g. [0.88,0.12][0.88, 0.12] — the decoder attends heavily to "Hola".
  • The context vector is c=0.88⋅VHola+0.12⋅VEOSc = 0.88 \cdot V_{\text{Hola}} + 0.12 \cdot V_{\text{EOS}}.
  • This context is passed to the FFN to produce the hidden state that is projected to predict "Hello".

Cross-attention is the encoder–decoder bridge

In encoder–decoder models every decoder layer has three sub-layers: (1) causal self-attention over what has been generated so far, (2) cross-attention over the encoder output, (3) the feed-forward network. If you remove the encoder and the cross-attention you get a decoder-only model (GPT style).

Multi-Head Attention variants

The KV cache problem

During autoregressive inference, the model generates one token at a time. To avoid recomputing keys and values for all previous tokens at each step, frameworks maintain a KV cache: a buffer storing KK and VV for every past position, for every layer.

KV cache size per layer:

KV cache=2×N×h×dk×bytes per element\text{KV cache} = 2 \times N \times h \times d_k \times \text{bytes per element}

where NN is the current sequence length and hh is the number of heads. For a LLaMA-2-70B-scale model with full MHA: h=64h = 64 heads, dk=128d_k = 128, N=4096N = 4096 tokens, bfloat16 (2 bytes) → 2×4096×64×128×2=134,217,7282 \times 4096 \times 64 \times 128 \times 2 = 134{,}217{,}728 bytes ≈ 134 MB per layer, times 80 layers ≈ 10.7 GB just for the KV cache at max context — per sequence being served. Multi-query and grouped-query attention attack this directly.

Multi-Query Attention (MQA)

In MQA there is a single shared KK and VV head across all hh query heads. Each query head ii still has its own WQiW_{Q_i}, but all heads share one WKW_K and one WVW_V:

Qi=X WQi,K=X WK,V=X WV∀iQ_i = X\,W_{Q_i}, \quad K = X\,W_K, \quad V = X\,W_V \quad \forall i

KV cache shrinks by a factor of hh: instead of storing hh copies of KK and VV, you store one. Quality drops slightly; true MQA is rare in practice — most modern LLMs prefer GQA instead.

Grouped-Query Attention (GQA)

GQA generalizes MQA: split the hh query heads into GG groups (1≤G≤h1 \le G \le h). Each group shares one KK and one VV head. G=1G = 1 is MQA; G=hG = h is standard MHA.

GQA with G groups:hQ=h,  hK=hV=G\text{GQA with } G \text{ groups:} \quad h_Q = h,\; h_K = h_V = G

Worked example: 8 heads, GQA with G = 2

  • 8 Q-heads (independent WQ1,…,WQ8W_{Q_1}, \ldots, W_{Q_8})
  • 2 K-heads, 2 V-heads (shared within each group of 4)
  • KV cache = 2×N×2×dk2 \times N \times 2 \times d_k — 4× smaller than MHA
VariantQ headsK/V headsKV cache factorNotes
MHAhhhh1×1\timesOriginal
GQAhhGGh/G×h/G\times smallerMistral, LLaMA-2/3
MQAhh11h×h\times smallerExtreme; PaLM, Falcon

GQA in practice

Mistral 7B uses GQA with h=32h=32 query heads and G=8G=8 KV heads — a 4× KV cache reduction. LLaMA-2 70B and LLaMA-3 70B both use GQA (h=64h=64 query heads, G=8G=8 KV heads) — an 8× cache reduction. Quality comparisons show GQA with G≈h/4G \approx h/4 to h/8h/8 is nearly indistinguishable from full MHA.

Rotary Positional Encoding (RoPE)

The idea

Instead of adding a sinusoidal vector to the token embedding (as the original Transformer does), RoPE encodes position by rotating the query and key vectors before computing dot products. No extra embedding is needed; position is injected directly into the attention score.

The rotation

For a query vector q∈Rd\mathbf{q} \in \mathbb{R}^d, group pairs of dimensions (q2i,q2i+1)(q_{2i}, q_{2i+1}) and rotate them by angle mθim \theta_i where:

  • mm is the absolute position of the token in the sequence
  • θi=base−2i/d\theta_i = \text{base}^{-2i/d}, with base=10000\text{base} = 10000 (following the sinusoidal PE convention)
RoPE(q,m)2i, 2i+1=(cos⁡(mθi)−sin⁡(mθi)sin⁡(mθi)−cos⁡(mθi))(q2iq2i+1)\text{RoPE}(\mathbf{q}, m)_{2i,\,2i+1} = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \phantom{-}\cos(m\theta_i) \end{pmatrix} \begin{pmatrix} q_{2i} \\ q_{2i+1} \end{pmatrix}

The same rotation is applied to k\mathbf{k} at position nn.

Key property: relative distance

Rotating q\mathbf{q} by angle mθim\theta_i and k\mathbf{k} by angle nθin\theta_i, the dot product of each pair picks up cos⁡(mθi)cos⁡(nθi)+sin⁡(mθi)sin⁡(nθi)=cos⁡((m−n)θi)\cos(m\theta_i)\cos(n\theta_i)+\sin(m\theta_i)\sin(n\theta_i) = \cos((m-n)\theta_i) — the rotation addition identity. The absolute positions mm and nn cancel; only the offset m−nm-n remains. The same applies across all dimension pairs simultaneously.

When you compute the dot product of a rotated query at position mm with a rotated key at position nn:

RoPE(q,m)⊤⋅RoPE(k,n)=q⊤R(m−n) k\text{RoPE}(\mathbf{q}, m)^\top \cdot \text{RoPE}(\mathbf{k}, n) = \mathbf{q}^\top R(m-n)\, \mathbf{k}

where R(m−n)R(m-n) is a rotation matrix depending only on the relative offset m−nm - n. This means attention scores are automatically sensitive to how far apart tokens are, not where they are in absolute terms — a property that improves generalization to sequence lengths not seen during training.

Python
import numpy as np

def rope_rotate_pair(q2i, q2i1, m, theta_i):
    """Rotate a pair of query dimensions (q_{2i}, q_{2i+1}) at position m."""
    cos_t = np.cos(m * theta_i)
    sin_t = np.sin(m * theta_i)
    return (cos_t * q2i - sin_t * q2i1,
            sin_t * q2i + cos_t * q2i1)

d, base = 8, 10000
thetas = [base ** (-2 * i / d) for i in range(d // 2)]  # θ_0 … θ_{d/2-1}

q = np.array([1.0, 0.5, -0.3, 0.8, 0.1, -0.2, 0.6, 0.4])
m = 3  # token at position 3

q_rot = np.zeros_like(q)
for i, theta_i in enumerate(thetas):
    q_rot[2*i], q_rot[2*i+1] = rope_rotate_pair(q[2*i], q[2*i+1], m, theta_i)

print("original q:", q)
print("rotated  q:", q_rot.round(4))

RoPE is used everywhere modern

RoPE was introduced by Su et al. (2021) and has since been adopted by LLaMA (all versions), Mistral, GPT-NeoX, Falcon, and Qwen. Its relative-distance property also makes it easier to extend context beyond the training length via techniques such as YaRN and NTK-aware scaling.

FlashAttention

The memory wall in standard attention

Standard attention:

Attn(Q,K,V)=softmax ⁣(QK⊤dk)V\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

requires materializing the N×NN \times N attention score matrix in full. For N=8192N = 8192 tokens and float16, that is 81922×2≈1288192^2 \times 2 \approx 128 MB per head per layer — stored in HBM (the GPU's slow main memory). With multiple heads and layers, this becomes prohibitive and requires many slow round-trips between HBM and the GPU's fast on-chip SRAM.

Tiled computation

FlashAttention (Dao et al., 2022) reorders the computation to never store the full N×NN \times N matrix:

  1. Split QQ, KK, and VV into blocks that fit in SRAM (fast on-chip memory).
  2. For each block of QQ, iterate over all blocks of KK and VV, computing partial softmax values and accumulating the weighted sum of VV on-chip.
  3. Use the online softmax trick to maintain a running max and normalizer so the partial results are numerically stable without seeing all scores first.

Result: the full N×NN \times N matrix is never written to HBM. Memory scales as O(N)O(N) instead of O(N2)O(N^2). The math is exactly equivalent to standard attention — no approximation.

MetricStandard attentionFlashAttention
HBM memoryO(N2)O(N^2)O(N)O(N)
ApproximationExactExact
Speedup1×2–4× (practical)
Key idea—Tile + online softmax

Why fewer HBM reads wins

On modern GPUs, compute (FLOPs) is cheap; bandwidth (reading/writing HBM) is expensive. Standard attention is memory-bandwidth bound, not compute bound. FlashAttention reads each element of QQ, KK, VV only once per pass, replacing many slow HBM accesses with fast SRAM operations — that is where the 2–4× wall-clock speedup comes from.

Watch the online softmax run

FlashAttention — streaming a softmax you cannot see all of

1/7

Standard attention materialises the whole N×N score matrix before it can take a softmax, because a softmax needs its denominator — which needs every score. At N = 8192 in float16 that is 128 MB **per head per layer**, and it scales quadratically: N = 131,072 needs 32.0 GB for a single head. The whole of FlashAttention follows from refusing to build that matrix, so the question is how to normalise a softmax you have not finished seeing.

python
1# per query block: m = -inf, l = 0, O = 0
2
3for j in range(n_key_blocks):
4 S = Q_i @ K_j.T / sqrt(d)
5
6 # the running max moves
7 m_new = max(m, S.max())
8
9 # correct everything already
10 # accumulated, by one factor
11 rescale = exp(m - m_new)
12 P = exp(S - m_new)
13
14 l = rescale * l + P.sum()
15 O = rescale * O + P @ V_j
16 m = m_new
17
18return O / l
state
score-matrix memory, float16, one head one layer
NN² scoresstandardtiled (this trace's blocks)
1,0241.0e+62 MB24 KB
4,0961.7e+732 MB96 KB
8,1926.7e+7128 MB192 KB
32,7681.1e+92.0 GB768 KB
131,0721.7e+1032.0 GB3072 KB

The tiled column is not a smaller constant on the same curve — it is a different curve. Scores live only inside one B_r × B_c tile; everything else that grows with N is Q, K, V themselves, which are O(N·d) and had to exist anyway.

A 12-key attention row computed twice: once the standard way, materialising every score, and once in three tiles that never hold more than four. Watch the rescale factor in block 1, where a larger score arrives, the running maximum moves, and one scalar exp(m_old − m_new) corrects every partial result accumulated so far — that single factor is what makes a streamed softmax possible. Then two measured payoffs: the page's claim of exact equivalence holds to the last bit of float64 across all 48 output components, and deleting the running max fails not gradually but completely, at exactly ln(MAX_FLOAT64) = 709.78.

What to notice:

  • Frame 4: the rescale is the whole trick. Block 1 contains a score larger than anything seen so far, so the running maximum moves from 0.634 to 0.862. Everything accumulated from block 0 was exponentiated against the old maximum and is now wrong by exactly exp⁡(mold−mnew)=0.7957\exp(m_{\text{old}} - m_{\text{new}}) = 0.7957. One scalar repairs an arbitrary amount of history — that is what lets a softmax be computed without ever seeing all its inputs.
  • Frame 5: a rescale of exactly 1. The last block's maximum is below the running one, so nothing needs correcting. The rescale is not a per-block cost; it fires only when the maximum actually moves.
  • Frame 6: "exactly equivalent" is testable, and it passes. Across all 48 output components the largest difference from standard attention is 2.2×10−162.2\times10^{-16}, i.e. float64 rounding. FlashAttention is not a fast approximation like sparse or low-rank attention — it computes the same function by a different association of the same arithmetic. Any output difference is a bug, never a tuning knob.
  • Frame 7: why step 3 says "numerically stable". Delete the running max and the method is algebraically identical and practically dead: finite at a max score of 704.1, NaN at 751.0. The threshold is not empirical — ln⁡(MAX_FLOAT64)=709.78\ln(\text{MAX\_FLOAT64}) = 709.78 falls exactly between those rows. Above it exp⁡\exp returns Inf, and Inf/Inf is NaN.

That last failure is worth remembering because it hides well: trained networks rarely produce scores near 709, so the bug survives review and then surfaces on a long context, an untrained model, or a lower-precision format — as NaN rather than as a slightly wrong number.

FlashAttention-2 (2023) and FlashAttention-3 (2024) push further with better work partitioning across thread blocks and tensor-core utilization, but the core idea (tile and keep data in SRAM) is unchanged.

Sliding-window and sparse attention

The quadratic wall

Full self-attention is O(N2)O(N^2) in both time and memory. For N=128,000N = 128{,}000 tokens (Claude's context), that is 128,0002=16.4128{,}000^2 = 16.4 billion attention scores per layer — even with FlashAttention's O(N)O(N) memory, the computation is still O(N2)O(N^2) FLOPs.

Sliding-window attention

Each token attends only to the ww nearest tokens (a window of size ww centered at itself, one-sided for causal models):

score(i,j)={Qi⋅Kj/dk∣i−j∣≤w/2−∞otherwise\text{score}(i, j) = \begin{cases} Q_i \cdot K_j / \sqrt{d_k} & |i - j| \le w/2 \\ -\infty & \text{otherwise} \end{cases}

Complexity: O(N⋅w)O(N \cdot w) instead of O(N2)O(N^2). With w=4096w = 4096 and N=128,000N = 128{,}000 this is a 31× reduction.

Local + global tokens

Pure sliding-window loses long-range coherence — position 0 cannot directly influence position N−1N-1. Two common fixes:

  • Longformer / BigBird: a handful of designated "global" tokens (e.g., [CLS]) attend to and receive attention from all positions, carrying long-range information.
  • Mistral 7B: uses sliding-window attention in lower layers (cheap local features) and full attention in upper layers (long-range integration).

Local vs global coherence trade-off

Sliding-window limits each token's receptive field to ww tokens. Information from far-away tokens reaches position ii only by propagating through many layers — the same limitation RNNs had. For tasks requiring tight long-range dependencies (complex reasoning, document summarization) pure local attention may degrade quality; combining local + global tokens mitigates this.

RMSNorm and SwiGLU

Modern LLMs swap out two components of the original Transformer: LayerNorm becomes RMSNorm and the ReLU FFN becomes SwiGLU. Both are small but consistent wins.

RMSNorm

Standard LayerNorm centers and scales:

LayerNorm(x)=γ⊙x−μσ2+ϵ+β\text{LayerNorm}(\mathbf{x}) = \boldsymbol{\gamma} \odot \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \boldsymbol{\beta}

RMSNorm drops the mean-centering step entirely, normalizing only by the root mean square:

RMS(x)=1d∑i=1dxi2,RMSNorm(x)=γ⊙xRMS(x)\text{RMS}(\mathbf{x}) = \sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2}, \qquad \text{RMSNorm}(\mathbf{x}) = \boldsymbol{\gamma} \odot \frac{\mathbf{x}}{\text{RMS}(\mathbf{x})}

No β\boldsymbol{\beta} parameter (the mean shift is dropped). Fewer operations, no mean subtraction, empirically equivalent quality. Used by LLaMA, T5, PaLM.

SwiGLU

The original FFN uses two weight matrices and ReLU:

FFNReLU(x)=ReLU(W1x+b1) W2+b2\text{FFN}_{\text{ReLU}}(\mathbf{x}) = \text{ReLU}(W_1 \mathbf{x} + \mathbf{b}_1)\,W_2 + \mathbf{b}_2

SwiGLU replaces this with a gated structure using three weight matrices:

SwiGLU(x)=(W1x⊙SiLU(W3x)) W2\text{SwiGLU}(\mathbf{x}) = \big(W_1 \mathbf{x} \odot \text{SiLU}(W_3 \mathbf{x})\big)\,W_2

where SiLU(x)=x⋅σ(x)=x/(1+e−x)\text{SiLU}(x) = x \cdot \sigma(x) = x / (1 + e^{-x}) (also called Swish). The gating term SiLU(W3x)\text{SiLU}(W_3 \mathbf{x}) acts as a smooth, learned filter on the features produced by W1xW_1 \mathbf{x}.

Python
import numpy as np

def silu(x):
    return x / (1 + np.exp(-x))   # x * sigmoid(x)

def swiglu(x, W1, W3, W2):
    gate = silu(x @ W3)            # gating signal
    hidden = (x @ W1) * gate       # element-wise gating
    return hidden @ W2             # project back

# Tiny example: d_model=4, d_ff=8 (usually 8/3 * d_model to match param count)
rng = np.random.RandomState(0)
d_model, d_ff = 4, 8
W1 = rng.randn(d_model, d_ff) * 0.1
W3 = rng.randn(d_model, d_ff) * 0.1
W2 = rng.randn(d_ff, d_model) * 0.1
x = rng.randn(3, d_model)          # 3 tokens
out = swiglu(x, W1, W3, W2)
print("SwiGLU output shape:", out.shape)  # (3, 4) — same as input

SwiGLU's extra weight matrix

SwiGLU adds a third projection matrix W3W_3 compared to the standard FFN. To keep total parameter count the same, the hidden dimension is reduced: instead of dff=4 dmodeld_{ff} = 4\,d_{\text{model}}, SwiGLU uses dff=83 dmodeld_{ff} = \tfrac{8}{3}\,d_{\text{model}} (rounded up to a multiple of 64 for hardware efficiency). LLaMA uses this ratio throughout.

Exercise

In encoder-decoder cross-attention (e.g., machine translation), where do the Queries, Keys, and Values come from?

Need a hint?

The decoder wants to 'look up' information from the encoder's output.

Exercise

A model with 16 Q-heads uses Grouped-Query Attention (GQA) with 4 KV groups. Compared to Multi-Head Attention (MHA), how large is the KV cache?

Need a hint?

KV cache scales with the number of K/V heads, not Q heads.

Exercise

What is the key advantage of Rotary Positional Encoding (RoPE) over additive sinusoidal encodings?

Need a hint?

Think about how RoPE encodes position and what property the dot product has.

Common mistakes

  1. Cross-attention Q/K/V source confusion: Q comes from the decoder; K and V come from the encoder. Reversing this makes the decoder unable to query the source.
  2. Forgetting the causal mask in decoder self-attention: the decoder's self-attention still needs a causal mask; only the cross-attention step attends to the full encoder output without a causal constraint.
  3. Expecting MQA/GQA to improve quality: these are inference optimizations. They reduce KV cache size at a small quality cost; they don't improve perplexity and shouldn't be chosen for quality reasons.
  4. Thinking FlashAttention changes the result: FlashAttention is numerically identical to standard attention. If outputs differ, it's a precision or implementation bug, not an intentional trade-off.
  5. Applying RoPE only to Q: RoPE must be applied to both QQ and KK for the relative-distance property to hold. Rotating only Q breaks the math.
  6. Confusing SwiGLU's dffd_{ff}: the SwiGLU FFN uses dff≈83 dmodeld_{ff} \approx \tfrac{8}{3}\,d_{\text{model}} (not 4×4\times) to match parameter counts with the two-matrix ReLU FFN.

Key takeaways

  • Cross-attention bridges encoder and decoder: Q from decoder, K/V from encoder; no causal mask on the cross-attention step.
  • MQA/GQA reduce KV cache memory at inference by sharing K/V heads across query head groups — GQA with G=h/4G = h/4 is the current sweet spot.
  • RoPE injects relative position into attention scores by rotating Q and K vectors; the dot product then depends only on position difference, not absolute position.
  • FlashAttention tiles the attention computation to avoid materializing the N×NN \times N matrix, achieving exact attention with O(N)O(N) memory and 2–4× wall-clock speedup.
  • Sliding-window attention reduces attention complexity from O(N2)O(N^2) to O(N⋅w)O(N \cdot w) for long contexts; global tokens preserve long-range connectivity.
  • RMSNorm drops mean-centering from LayerNorm; SwiGLU adds learned gating to the FFN — both are small, consistent quality improvements used in all major modern LLMs.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Modern Attention MechanismsRecurrent Neural NetworksCalculus for MLState Space Models (S4 & …The Transformer Architect…Multi-Head Attention & Po…Self-AttentionFoundation Models & Scali…Attention Mechanisms: Sel…Positional Encodings: Sin…