Multi-Head Attention & Positional Encoding
A single attention head learns one notion of relevance. But language has many simultaneous relationships — syntax, coreference, tense, topic. Multi-head attention runs several attention computations in parallel so the model can attend to different things at once. And because attention has no built-in sense of order, we inject position explicitly with positional encodings.
One head sees one thing
A single self-attention head produces one set of attention weights — one "view" of how tokens relate. Forcing every relationship through one head is limiting: the head that tracks "which adjective modifies which noun" is not the head that tracks "which pronoun refers to which subject."
Intuition
Think of multi-head attention as a committee of specialists reading the same sentence. One head watches subject–verb agreement, another tracks pronoun references, another follows the topic. Each reports back, and the model combines their findings into a richer representation than any single reader could produce.
Multi-head attention
Split the model dimension across heads, each of size . Each head has its own , runs scaled dot-product attention independently, then all heads are concatenated and mixed by an output matrix :
Note the cost is roughly the same as one full-size head: each head is as wide, so total compute is unchanged — you get diversity for free.
Worked example: dimension bookkeeping
GPT-style block: , heads, sequence length . Each head uses
Track the shapes for one head (the input is ):
| Quantity | Multiply | Shape |
|---|---|---|
| — | ||
| per-head projections | ||
| same | ||
| scores | ||
Eight heads each output . Concatenating along the feature axis gives
back to . Finally is , so the output is — same shape as the input. In practice the per-head are stacked into one matrix and reshaped, which is why the parameter count below uses full matrices.
Worked example: parameter count
Multi-head attention has exactly four learned projections, each : the stacked and the output . Ignoring biases,
Crucially this is independent of : splitting into more heads only reshapes the same matrices, so 1 head and 8 heads cost the same parameters and (nearly) the same FLOPs. You get multiple attention views for free.
import torch
import torch.nn as nn
mha = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
x = torch.randn(2, 10, 512) # (batch, seq, d_model)
out, attn_weights = mha(x, x, x) # self-attention: Q=K=V=x
print(out.shape) # (2, 10, 512)
print(attn_weights.shape) # (2, 10, 10) — averaged over heads
# From scratch, to see the reshape into heads:
def multi_head_attention(X, Wq, Wk, Wv, Wo, num_heads):
B, T, D = X.shape
d_k = D // num_heads
Q = (X @ Wq).view(B, T, num_heads, d_k).transpose(1, 2) # (B, h, T, d_k)
K = (X @ Wk).view(B, T, num_heads, d_k).transpose(1, 2)
V = (X @ Wv).view(B, T, num_heads, d_k).transpose(1, 2)
scores = (Q @ K.transpose(-2, -1)) / d_k**0.5
attn = scores.softmax(dim=-1)
out = (attn @ V).transpose(1, 2).reshape(B, T, D) # concat heads
return out @ Wo
What heads actually learn
Probing trained models shows specialized heads: some attend to the previous token, some to the syntactic head of a phrase, some to rare/important words. Not every head is useful — many can be pruned after training with little accuracy loss.
The order problem
Self-attention is permutation-invariant: shuffle the input tokens and the set of outputs is just shuffled the same way — the values are unchanged. The formula contains no notion of position. But "dog bites man" ≠ "man bites dog." We must inject order information.
Positional encoding
The original Transformer adds a fixed sinusoidal vector to each token embedding, where each dimension oscillates at a different frequency:
- — position in the sequence (0, 1, 2, …)
- — dimension index, giving wavelengths from up to
Why sinusoids? Low frequencies encode coarse position, high frequencies encode fine position — like the hands of a clock.
Sinusoidal positional encoding
Cosine similarity to pos 3
Each row is a position's encoding vector. The heatmap colour maps PE values from -1 (blue) through 0 (light) to +1 (indigo). High-frequency dimensions (left) vary rapidly between adjacent positions; low-frequency dimensions (right) change slowly. The bar chart shows how similar the selected position's vector is to every other position — nearby positions are more similar.
Worked example: computing PE by hand
Take a tiny model dimension , so there are two frequency pairs: uses dimensions and uses dimensions . The two angular frequencies are
Now fill in a few positions ( in radians):
| dim 0 = | dim 1 = | dim 2 = | dim 3 = | |
|---|---|---|---|---|
| 0 | ||||
| 1 | ||||
| 2 |
Notice the high-frequency pair (dims 0–1) swings a lot between adjacent positions, while the low-frequency pair (dims 2–3) barely moves — exactly the coarse/fine split. The notebook reproduces every number in this table.
Why this captures relative position
Because obey the angle-addition identities, shifting position by a fixed offset is a fixed linear rotation of the encoding — independent of the absolute position . For the frequency of one pair, write . Then
i.e. where is the rotation by . The offset matrix depends only on , not on , so the model can learn to attend "three tokens back" with a single fixed transform that works everywhere in the sequence. It also extrapolates to sequence lengths longer than seen in training.
import numpy as np
def positional_encoding(seq_len, d_model):
pos = np.arange(seq_len)[:, None]
i = np.arange(d_model)[None, :]
angle = pos / np.power(10000, (2 * (i // 2)) / d_model)
pe = np.zeros((seq_len, d_model))
pe[:, 0::2] = np.sin(angle[:, 0::2])
pe[:, 1::2] = np.cos(angle[:, 1::2])
return pe
pe = positional_encoding(50, 128)
print(pe.shape) # (50, 128) — added directly to token embeddings
Learned and rotary alternatives
| Method | How | Used by |
|---|---|---|
| Sinusoidal | Fixed sin/cos | Original Transformer |
| Learned absolute | Trainable position embeddings | BERT, GPT-2 |
| RoPE (rotary) | Rotate Q/K by position-dependent angle | LLaMA, most modern LLMs |
| ALiBi | Bias attention scores by distance | Some long-context models |
Modern LLMs mostly use RoPE, which encodes relative position directly inside the attention dot product and extrapolates better to long contexts.
Don't skip positional information
Without positional encoding a Transformer is a bag-of-words — it literally cannot tell "dog bites man" from "man bites dog." This is the most common bug when implementing a Transformer from scratch.
Exercise
A Transformer has d_model = 512 and 8 attention heads. What is the dimension d_k of each head?
Need a hint?
Split the model dimension evenly across heads.
Exercise
Why do Transformers need positional encoding?
Need a hint?
What happens to the attention output if you shuffle the input tokens?
Exercise
Without positional encoding, a Transformer effectively treats the input as what?
Need a hint?
Order doesn't matter to it.
Common mistakes
- Heads that don't divide the dimension: must be divisible by (e.g. 512/8 = 64). Otherwise the reshape fails.
- Forgetting positional encoding: silently turns the model into a permutation-invariant bag-of-words.
- Adding vs concatenating PE: sinusoidal/learned PE is added to embeddings (same dimension), not concatenated.
Key takeaways
- Multi-head attention runs several attention views in parallel, then concatenates.
- Each head uses , so total compute matches one full-size head.
- Self-attention is permutation-invariant — position must be injected.
- Positional encodings (sinusoidal, learned, or RoPE) supply word order.
Related concepts
- Self-Attention: the single-head mechanism this scales up
- The Transformer Architecture: assembling these into full blocks