Self-Attention
RNNs process a sequence step by step, squeezing all history into a single hidden state. Self-attention throws that bottleneck away: every token directly looks at every other token and decides how much to borrow from each. This one idea ("attention is all you need") powers GPT, BERT, and essentially every modern language model. As a hypothesis-space choice it is the least-biased of the four architectures in the inductive-bias story: it assumes only permutation symmetry, which is exactly why it needs positional encodings (next lesson) and why, given enough data, it overtakes the more heavily-biased CNN and RNN.
Real-world examples: Large language models (GPT, Claude, Llama), machine translation, protein folding (AlphaFold), vision (ViT), code completion.
The problem with recurrence
An RNN connecting word 1 to word 100 must pass information through 99 sequential steps — slow to train (no parallelism) and lossy (vanishing gradients). Self-attention connects any two positions in a single step, and computes all positions in parallel.
Intuition
Reading "The animal didn't cross the street because it was tired," you instantly know "it" = "animal," not "street." Self-attention lets the model do the same: when encoding "it," it attends strongly to "animal." Each word builds its meaning from the words it finds relevant — looked up by content, not by position.
Query, Key, Value
Self-attention reframes each token as a tiny database lookup. From each token's embedding we project three vectors via learned matrices :
- Query — "what am I looking for?"
- Key — "what do I offer?"
- Value — "what I'll actually hand over"
A token's query is compared against every key to score relevance; those scores weight the values that get summed up.
The library analogy
Your query is the topic you want. Every book has a key printed on its spine (its subject) and value content inside. You match your query against all spines, then read the books in proportion to how well their spine matched — a little from many books, a lot from the best one.
Self-attention weights = softmax(QKᵀ / √d)
Each row is one query token's attention over all keys; the row sums to 1 (a softmax). Brighter = more attention. Tokens attend most to tokens whose embeddings align with theirs.
Scaled dot-product attention
The whole mechanism is one formula:
Step by step:
- Scores — dot product of every query with every key. High dot product = high relevance.
- Scale by — keeps scores from growing with dimension (see below).
- Softmax: turn each row of scores into weights that sum to 1 (the attention weights).
- Weighted sum: multiply weights by : each output is a blend of all values, weighted by relevance.
Why divide by ?
Without scaling, dot-product scores grow in magnitude with dimension — when components are unit-variance. For large this pushes softmax into its saturated region (one weight ≈ 1, the rest ≈ 0), killing gradients. Dividing by restores unit variance and keeps softmax well-behaved regardless of dimension.
For the full variance derivation and a complete 3-token hand-worked example (scores → scale → softmax weights → output, all numbers shown), see:
In PyTorch
import torch
import torch.nn.functional as F
X = torch.randn(1, 4, 8) # (batch, seq, d_model)
Wq = torch.randn(8, 8); Wk = torch.randn(8, 8); Wv = torch.randn(8, 8)
Q, K, V = X @ Wq, X @ Wk, X @ Wv
# PyTorch 2.0+ fused, numerically stable attention:
out = F.scaled_dot_product_attention(Q, K, V)
print(out.shape) # (1, 4, 8)
Causal (masked) attention
In a language model predicting the next token, position must not peek at future positions. We enforce this by adding to scores for future positions before the softmax, so their weights become 0:
seq = 4
mask = torch.triu(torch.ones(seq, seq), diagonal=1).bool() # upper triangle
scores = scores.masked_fill(mask, float('-inf')) # future → -inf
This causal mask is what makes GPT-style autoregressive generation possible.
Why attention won
| RNN | Self-Attention | |
|---|---|---|
| Path length between any two tokens | ||
| Parallelizable across sequence? | No | Yes |
| Long-range dependencies | Hard (vanishing) | Direct |
| Compute per layer |
The cost: attention is quadratic in sequence length ( scores). For very long sequences that becomes the bottleneck — hence sparse/linear-attention research. But the constant path length and full parallelism made it the clear winner for training at scale.
Exercise
In self-attention, which vector represents 'what a token is looking for'?
Need a hint?
Query, Key, or Value?
Exercise
Why does scaled dot-product attention divide the scores by √d_k?
Need a hint?
Think about how the variance of a dot product grows with dimension.
Exercise
What is the computational complexity of self-attention in sequence length n?
Need a hint?
Every token attends to every other token.
Common mistakes
- Forgetting the scale factor — softmax saturates, gradients vanish, training stalls.
- Omitting the causal mask in a generative model — the model "cheats" by seeing the answer, then fails at inference.
- Confusing the dimensions: attention weights are (token-to-token), the output is (same shape as input).
Key takeaways
- Self-attention lets every token attend to every other in one parallel step.
- Query·Key scores, scaled by and softmaxed, weight the Values.
- A causal mask enables autoregressive (GPT-style) generation.
- Cost is quadratic in sequence length — the main scaling bottleneck.
Related concepts
- Multi-Head Attention & Positional Encoding: many attention heads + injecting word order
- LSTM & GRU: the recurrent approach attention replaced
- Inductive Bias & Equivariance: why attention is the "keep only permutation symmetry, inject position" move, and the least-biased of the four architectures