Attention Mechanisms: Self, Cross, Causal, MHA, MQA, and GQA
All attention variants share the same engine. They differ in where Q, K, V come from and what tokens are visible to each query position.
The base operation
(queries), (keys), (values). Each row of the output is a weighted blend of value rows, where the weights encode relevance.
Taxonomy
| Variant | Q source | K, V source | Mask |
|---|---|---|---|
| Self-attention | Sequence itself | Same sequence | None (bidirectional) |
| Causal self-attention | Sequence itself | Same sequence | Lower-triangular |
| Cross-attention | Decoder state | Encoder output | None (over encoder) |
| Multi-head (MHA) | projections | projections | Depends on context |
| Multi-query (MQA) | projections | 1 shared K/V head | — |
| Grouped-query (GQA) | projections in groups | shared K/V heads | — |
Self-attention
The same input sequence is projected to all three roles:
Score measures how much token should borrow from token . After softmax, each output token is a context-weighted blend of all value rows — giving every token access to global context in a single parallel operation.
Self-attention in an encoder (e.g., BERT) is bidirectional: no mask, so every token can see every other token. This is ideal for understanding tasks where the full input is available at once.
Causal (masked) self-attention
A decoder generating tokens autoregressively cannot see future positions. The mask is applied before softmax by adding to positions where :
After softmax, , so masked positions contribute nothing. The computation stays fully batched — no variable-length loops.
The mask is just a matrix add
The causal mask does not require separate code paths or ragged tensors. A static lower-triangular boolean matrix is broadcast across the batch and head dimensions. Adding before softmax handles it in one fused kernel.
Cross-attention
Cross-attention connects two different sequences — the decoder's evolving state () and the encoder's fixed output ():
For each decoder position, the query looks up which encoder positions are relevant (via ) and blends their value vectors. Because and are fixed for an entire decoding pass, they can be pre-computed and cached — cross-attention inference is cheap.
Decoder-only vs encoder-decoder: Remove the encoder and the cross-attention sub-layer and you get GPT-style decoder-only models. Encoder-decoder models can attend bidirectionally over the source; decoder-only models rely on the causal self-attention stack to process the prefix in context.
Multi-Head Attention (MHA)
Running one attention head gives one view of relevance. MHA runs independent heads in parallel, each projecting to dimension , then concatenating:
Different heads specialize: one may track syntactic dependencies, another coreference, another positional proximity. Total parameter count stays the same as one large head ( are smaller by factor , but there are of them).
Complexity: compute, activation memory for the attention score matrices. The term is the scaling bottleneck for long sequences.
Multi-Query Attention (MQA)
MQA (Shazeer 2019) shares a single K/V head across all query heads:
Memory win: MHA stores K/V matrices per layer in the KV cache; MQA stores 1 — a factor-of- reduction. For LLaMA-2 70B with heads, that is 64× smaller. The trade-off is a modest quality drop: one K/V head has less capacity to represent diverse key structures.
Grouped-Query Attention (GQA)
GQA (Ainslie et al. 2023) interpolates between MHA () and MQA ():
- Split the query heads into groups of size .
- Each group shares one K/V head — KV cache shrinks by .
- At , memory is smaller with only a small quality drop.
LLaMA-3, Mistral, Gemma, Falcon, and most modern open-weight LLMs use GQA. is a common default balancing cache savings vs quality.
KV cache comparison (per layer, tokens, , , bfloat16 — LLaMA-2 70B attention dims):
| Variant | KV heads | Cache per layer |
|---|---|---|
| MHA | 64 | 128 MB |
| GQA () | 8 | 16 MB |
| MQA | 1 | 2 MB |
FlashAttention
Standard attention materializes the full score matrix in GPU HBM (high-bandwidth memory). FlashAttention (Dao et al. 2022) tiles the computation to fit in the much-faster SRAM:
- No activation memory: peak memory is .
- Identical numerical result: not an approximation; it recomputes partial softmax statistics on the fly.
- 2–4× faster wall-clock on A100 for typical sequence lengths due to reduced HBM bandwidth.
FlashAttention-2 and -3 further optimize for modern tensor core layouts. In PyTorch, torch.nn.functional.scaled_dot_product_attention dispatches to FlashAttention automatically when available.
Sparse attention patterns
For very long sequences, is prohibitive even with FlashAttention. Sparse patterns restrict each query to a subset of keys:
- Sliding window (local): each position attends to its nearest neighbors. Used in Longformer, Mistral's sliding window.
- Global + local: a few designated tokens (e.g.,
[CLS]) attend globally; all others attend locally. Used in BigBird, Longformer. - Linear attention: approximate the softmax with a kernel trick to achieve — quality lower than exact attention.
Complexity summary:
| Pattern | Compute | Activation memory |
|---|---|---|
| Full (FlashAttn) | ||
| Sliding window | ||
| Linear (approx) |
Choosing the right variant
- Encoder for classification / understanding: bidirectional MHA (no mask).
- Decoder-only LLM (training): causal MHA; switch to GQA when deploying at scale.
- Translation / summarization (seq2seq): causal decoder self-attention + cross-attention over encoder.
- Long documents (): FlashAttention is a free upgrade; add sliding window for .
- Inference with large batches: prefer GQA — KV cache is the dominant memory cost.
Related concepts
- Scaled Dot-Product Attention: the derivation and a 3-token worked example
- Self-Attention: source lesson: QKV intuition and causal masking
- Modern Attention Mechanisms: lesson: cross-attention, MQA/GQA, RoPE, FlashAttention
- Positional Encodings: how tokens know their position in the sequence