Multi-Head Attention & Positional EncodingOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Multi-Head Attention & Positional Encoding

A single attention head learns one notion of relevance. But language has many simultaneous relationships — syntax, coreference, tense, topic. Multi-head attention runs several attention computations in parallel so the model can attend to different things at once. And because attention has no built-in sense of order, we inject position explicitly with positional encodings.

One head sees one thing

A single self-attention head produces one set of attention weights — one "view" of how tokens relate. Forcing every relationship through one head is limiting: the head that tracks "which adjective modifies which noun" is not the head that tracks "which pronoun refers to which subject."

Intuition

Think of multi-head attention as a committee of specialists reading the same sentence. One head watches subject–verb agreement, another tracks pronoun references, another follows the topic. Each reports back, and the model combines their findings into a richer representation than any single reader could produce.

Multi-head attention

Split the model dimension dmodeld_{\text{model}} across hh heads, each of size dk=dmodel/hd_k = d_{\text{model}}/h. Each head has its own WQ(i),WK(i),WV(i)W_Q^{(i)}, W_K^{(i)}, W_V^{(i)}, runs scaled dot-product attention independently, then all heads are concatenated and mixed by an output matrix WOW_O:

headi=Attention(QWQ(i), KWK(i), VWV(i))\text{head}_i = \text{Attention}(QW_Q^{(i)},\, KW_K^{(i)},\, VW_V^{(i)}) MultiHead(Q,K,V)=Concat(head1,…,headh) WO\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)\,W_O

Note the cost is roughly the same as one full-size head: each head is 1/h1/h as wide, so total compute is unchanged — you get diversity for free.

Worked example: dimension bookkeeping

GPT-style block: dmodel=512d_{\text{model}} = 512, h=8h = 8 heads, sequence length T=10T = 10. Each head uses

dk=dmodelh=5128=64.d_k = \frac{d_{\text{model}}}{h} = \frac{512}{8} = 64.

Track the shapes for one head ii (the input XX is 10×51210 \times 512):

QuantityMultiplyShape
XX—10×51210 \times 512
WQ(i),WK(i),WV(i)W_Q^{(i)},W_K^{(i)},W_V^{(i)}per-head projections512×64512 \times 64
Q(i)=XWQ(i)Q^{(i)}=XW_Q^{(i)}(10×512)(512×64)(10{\times}512)(512{\times}64)10×6410 \times 64
K(i),V(i)K^{(i)},V^{(i)}same10×6410 \times 64
scores Q(i)K(i)⊤/dkQ^{(i)}K^{(i)\top}/\sqrt{d_k}(10×64)(64×10)(10{\times}64)(64{\times}10)10×1010 \times 10
headi=softmax(⋅)V(i)\text{head}_i = \text{softmax}(\cdot)V^{(i)}(10×10)(10×64)(10{\times}10)(10{\times}64)10×6410 \times 64

Eight heads each output 10×6410 \times 64. Concatenating along the feature axis gives

Concat(head1,…,head8):10×(8⋅64)=10×512,\text{Concat}(\text{head}_1,\dots,\text{head}_8):\quad 10 \times (8 \cdot 64) = 10 \times 512,

back to dmodeld_{\text{model}}. Finally WOW_O is 512×512512 \times 512, so the output is (10×512)(512×512)=10×512(10{\times}512)(512{\times}512) = 10 \times 512 — same shape as the input. In practice the per-head WQ(i)W_Q^{(i)} are stacked into one 512×512512 \times 512 matrix WQW_Q and reshaped, which is why the parameter count below uses full dmodel×dmodeld_{\text{model}} \times d_{\text{model}} matrices.

Worked example: parameter count

Multi-head attention has exactly four learned projections, each dmodel×dmodeld_{\text{model}} \times d_{\text{model}}: the stacked WQ,WK,WVW_Q, W_K, W_V and the output WOW_O. Ignoring biases,

#params=4 dmodel2=4⋅5122=4⋅262,144=1,048,576≈1.05M.\#\text{params} = 4\,d_{\text{model}}^2 = 4 \cdot 512^2 = 4 \cdot 262{,}144 = 1{,}048{,}576 \approx 1.05\text{M}.

Crucially this is independent of hh: splitting into more heads only reshapes the same matrices, so 1 head and 8 heads cost the same parameters and (nearly) the same FLOPs. You get multiple attention views for free.

Python
import torch
import torch.nn as nn

mha = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
x = torch.randn(2, 10, 512)                       # (batch, seq, d_model)
out, attn_weights = mha(x, x, x)                  # self-attention: Q=K=V=x

print(out.shape)            # (2, 10, 512)
print(attn_weights.shape)   # (2, 10, 10) — averaged over heads
Python
# From scratch, to see the reshape into heads:
def multi_head_attention(X, Wq, Wk, Wv, Wo, num_heads):
    B, T, D = X.shape
    d_k = D // num_heads
    Q = (X @ Wq).view(B, T, num_heads, d_k).transpose(1, 2)   # (B, h, T, d_k)
    K = (X @ Wk).view(B, T, num_heads, d_k).transpose(1, 2)
    V = (X @ Wv).view(B, T, num_heads, d_k).transpose(1, 2)
    scores = (Q @ K.transpose(-2, -1)) / d_k**0.5
    attn = scores.softmax(dim=-1)
    out = (attn @ V).transpose(1, 2).reshape(B, T, D)          # concat heads
    return out @ Wo

What heads actually learn

Probing trained models shows specialized heads: some attend to the previous token, some to the syntactic head of a phrase, some to rare/important words. Not every head is useful — many can be pruned after training with little accuracy loss.

The order problem

Self-attention is permutation-invariant: shuffle the input tokens and the set of outputs is just shuffled the same way — the values are unchanged. The formula softmax(QK⊤/dk)V\text{softmax}(QK^\top/\sqrt{d_k})V contains no notion of position. But "dog bites man" ≠ "man bites dog." We must inject order information.

Positional encoding

The original Transformer adds a fixed sinusoidal vector to each token embedding, where each dimension oscillates at a different frequency:

PE(pos, 2i)=sin⁡ ⁣(pos100002i/d)PE(pos, 2i+1)=cos⁡ ⁣(pos100002i/d)PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right) \qquad PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)
  • pospos — position in the sequence (0, 1, 2, …)
  • ii — dimension index, giving wavelengths from 2π2\pi up to ∼ ⁣10000⋅2π\sim\!10000\cdot 2\pi

Why sinusoids? Low frequencies encode coarse position, high frequencies encode fine position — like the hands of a clock.

Sinusoidal positional encoding

Selected positionpos 3
Self-similarity1.000
Sim to pos ±10.957
Sim to pos ±40.729
01234567891011121314150481216202428dimension →

Cosine similarity to pos 3

0.00.51.00123456789101112131415
selected±1±2–3far

Each row is a position's encoding vector. The heatmap colour maps PE values from -1 (blue) through 0 (light) to +1 (indigo). High-frequency dimensions (left) vary rapidly between adjacent positions; low-frequency dimensions (right) change slowly. The bar chart shows how similar the selected position's vector is to every other position — nearby positions are more similar.

Worked example: computing PE by hand

Take a tiny model dimension d=4d = 4, so there are two frequency pairs: i=0i = 0 uses dimensions (0,1)(0,1) and i=1i = 1 uses dimensions (2,3)(2,3). The two angular frequencies are

1100000/4=11=1,1100002/4=110000=1100=0.01.\frac{1}{10000^{0/4}} = \frac{1}{1} = 1, \qquad \frac{1}{10000^{2/4}} = \frac{1}{\sqrt{10000}} = \frac{1}{100} = 0.01.

Now fill in a few positions (sin⁡/cos⁡\sin/\cos in radians):

posposdim 0 = sin⁡(pos⋅1)\sin(pos\cdot 1)dim 1 = cos⁡(pos⋅1)\cos(pos\cdot 1)dim 2 = sin⁡(pos⋅0.01)\sin(pos\cdot 0.01)dim 3 = cos⁡(pos⋅0.01)\cos(pos\cdot 0.01)
0sin⁡0=0\sin 0 = 0cos⁡0=1\cos 0 = 1sin⁡0=0\sin 0 = 0cos⁡0=1\cos 0 = 1
1sin⁡1=0.8415\sin 1 = 0.8415cos⁡1=0.5403\cos 1 = 0.5403sin⁡0.01=0.0100\sin 0.01 = 0.0100cos⁡0.01=1.0000\cos 0.01 = 1.0000
2sin⁡2=0.9093\sin 2 = 0.9093cos⁡2=−0.4161\cos 2 = -0.4161sin⁡0.02=0.0200\sin 0.02 = 0.0200cos⁡0.02=0.9998\cos 0.02 = 0.9998

Notice the high-frequency pair (dims 0–1) swings a lot between adjacent positions, while the low-frequency pair (dims 2–3) barely moves — exactly the coarse/fine split. The notebook reproduces every number in this table.

Why this captures relative position

Because sin⁡/cos⁡\sin/\cos obey the angle-addition identities, shifting position by a fixed offset kk is a fixed linear rotation of the encoding — independent of the absolute position pospos. For the frequency ω\omega of one pair, write p(pos)=[sin⁡(ω pos), cos⁡(ω pos)]\mathbf{p}(pos) = [\sin(\omega\,pos),\,\cos(\omega\,pos)]. Then

sin⁡(ω(pos+k))=sin⁡(ω pos)cos⁡(ωk)+cos⁡(ω pos)sin⁡(ωk),cos⁡(ω(pos+k))=cos⁡(ω pos)cos⁡(ωk)−sin⁡(ω pos)sin⁡(ωk),\begin{aligned} \sin(\omega(pos+k)) &= \sin(\omega\,pos)\cos(\omega k) + \cos(\omega\,pos)\sin(\omega k),\\ \cos(\omega(pos+k)) &= \cos(\omega\,pos)\cos(\omega k) - \sin(\omega\,pos)\sin(\omega k), \end{aligned}

i.e. p(pos+k)=R(ωk) p(pos)\mathbf{p}(pos+k) = R(\omega k)\,\mathbf{p}(pos) where RR is the 2×22\times 2 rotation by ωk\omega k. The offset matrix R(ωk)R(\omega k) depends only on kk, not on pospos, so the model can learn to attend "three tokens back" with a single fixed transform that works everywhere in the sequence. It also extrapolates to sequence lengths longer than seen in training.

Python
import numpy as np

def positional_encoding(seq_len, d_model):
    pos = np.arange(seq_len)[:, None]
    i = np.arange(d_model)[None, :]
    angle = pos / np.power(10000, (2 * (i // 2)) / d_model)
    pe = np.zeros((seq_len, d_model))
    pe[:, 0::2] = np.sin(angle[:, 0::2])
    pe[:, 1::2] = np.cos(angle[:, 1::2])
    return pe

pe = positional_encoding(50, 128)
print(pe.shape)             # (50, 128) — added directly to token embeddings

Learned and rotary alternatives

MethodHowUsed by
SinusoidalFixed sin/cosOriginal Transformer
Learned absoluteTrainable position embeddingsBERT, GPT-2
RoPE (rotary)Rotate Q/K by position-dependent angleLLaMA, most modern LLMs
ALiBiBias attention scores by distanceSome long-context models

Modern LLMs mostly use RoPE, which encodes relative position directly inside the attention dot product and extrapolates better to long contexts.

Don't skip positional information

Without positional encoding a Transformer is a bag-of-words — it literally cannot tell "dog bites man" from "man bites dog." This is the most common bug when implementing a Transformer from scratch.

Exercise

A Transformer has d_model = 512 and 8 attention heads. What is the dimension d_k of each head?

Need a hint?

Split the model dimension evenly across heads.

Exercise

Why do Transformers need positional encoding?

Need a hint?

What happens to the attention output if you shuffle the input tokens?

Exercise

Without positional encoding, a Transformer effectively treats the input as what?

Need a hint?

Order doesn't matter to it.

Common mistakes

  1. Heads that don't divide the dimension: dmodeld_{\text{model}} must be divisible by hh (e.g. 512/8 = 64). Otherwise the reshape fails.
  2. Forgetting positional encoding: silently turns the model into a permutation-invariant bag-of-words.
  3. Adding vs concatenating PE: sinusoidal/learned PE is added to embeddings (same dimension), not concatenated.

Key takeaways

  • Multi-head attention runs several attention views in parallel, then concatenates.
  • Each head uses dk=dmodel/hd_k = d_{model}/h, so total compute matches one full-size head.
  • Self-attention is permutation-invariant — position must be injected.
  • Positional encodings (sinusoidal, learned, or RoPE) supply word order.

Related concepts

Deep dive
Positional Encodings: Sinusoidal, RoPE, ALiBi, and Learned
Includes runnable notebook
Deep dive
Attention Mechanisms: Self, Cross, Causal, MHA, MQA, GQA
Includes runnable notebook

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Multi-Head Attention & Positional…Recurrent Neural NetworksCalculus for MLSelf-AttentionThe Transformer Architect…Modern Attention Mechanis…Positional Encodings: Sin…Attention Mechanisms: Sel…Neural Radiance Fields (N…Scaled Dot-Product Attent…