Self-AttentionOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Self-Attention

RNNs process a sequence step by step, squeezing all history into a single hidden state. Self-attention throws that bottleneck away: every token directly looks at every other token and decides how much to borrow from each. This one idea ("attention is all you need") powers GPT, BERT, and essentially every modern language model. As a hypothesis-space choice it is the least-biased of the four architectures in the inductive-bias story: it assumes only permutation symmetry, which is exactly why it needs positional encodings (next lesson) and why, given enough data, it overtakes the more heavily-biased CNN and RNN.

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: Large language models (GPT, Claude, Llama), machine translation, protein folding (AlphaFold), vision (ViT), code completion.

The problem with recurrence

An RNN connecting word 1 to word 100 must pass information through 99 sequential steps — slow to train (no parallelism) and lossy (vanishing gradients). Self-attention connects any two positions in a single step, and computes all positions in parallel.

Intuition

Reading "The animal didn't cross the street because it was tired," you instantly know "it" = "animal," not "street." Self-attention lets the model do the same: when encoding "it," it attends strongly to "animal." Each word builds its meaning from the words it finds relevant — looked up by content, not by position.

Query, Key, Value

Self-attention reframes each token as a tiny database lookup. From each token's embedding xi\mathbf{x}_i we project three vectors via learned matrices WQ,WK,WVW_Q, W_K, W_V:

  • Query qi=WQxi\mathbf{q}_i = W_Q\mathbf{x}_i — "what am I looking for?"
  • Key ki=WKxi\mathbf{k}_i = W_K\mathbf{x}_i — "what do I offer?"
  • Value vi=WVxi\mathbf{v}_i = W_V\mathbf{x}_i — "what I'll actually hand over"

A token's query is compared against every key to score relevance; those scores weight the values that get summed up.

The library analogy

Your query is the topic you want. Every book has a key printed on its spine (its subject) and value content inside. You match your query against all spines, then read the books in proportion to how well their spine matched — a little from many books, a lot from the best one.

Self-attention weights = softmax(QKᵀ / √d)

query:
ThecatsatonthematThecatsatonthematkeys →0.200.180.140.150.150.180.100.360.160.180.120.080.120.240.200.190.160.100.120.250.180.180.150.110.150.200.180.180.170.130.190.150.130.150.160.21
querycat
attends most tocat
row sum1.00

Each row is one query token's attention over all keys; the row sums to 1 (a softmax). Brighter = more attention. Tokens attend most to tokens whose embeddings align with theirs.

Scaled dot-product attention

The whole mechanism is one formula:

Attention(Q,K,V)=softmax ⁣(QK⊤dk)V\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V

Step by step:

  1. Scores QK⊤QK^{\top} — dot product of every query with every key. High dot product = high relevance.
  2. Scale by 1dk\frac{1}{\sqrt{d_k}} — keeps scores from growing with dimension (see below).
  3. Softmax: turn each row of scores into weights that sum to 1 (the attention weights).
  4. Weighted sum: multiply weights by VV: each output is a blend of all values, weighted by relevance.

Why divide by dk\sqrt{d_k}?

Without scaling, dot-product scores grow in magnitude with dimension — Var⁡(q⋅k)=dk\operatorname{Var}(\mathbf{q}\cdot\mathbf{k}) = d_k when components are unit-variance. For large dkd_k this pushes softmax into its saturated region (one weight ≈ 1, the rest ≈ 0), killing gradients. Dividing by dk\sqrt{d_k} restores unit variance and keeps softmax well-behaved regardless of dimension.

For the full variance derivation and a complete 3-token hand-worked example (scores → scale → softmax weights → output, all numbers shown), see:

Deep dive
Scaled Dot-Product Attention
Includes runnable notebook

In PyTorch

Python
import torch
import torch.nn.functional as F

X = torch.randn(1, 4, 8)                   # (batch, seq, d_model)
Wq = torch.randn(8, 8); Wk = torch.randn(8, 8); Wv = torch.randn(8, 8)
Q, K, V = X @ Wq, X @ Wk, X @ Wv

# PyTorch 2.0+ fused, numerically stable attention:
out = F.scaled_dot_product_attention(Q, K, V)
print(out.shape)                           # (1, 4, 8)

Causal (masked) attention

In a language model predicting the next token, position tt must not peek at future positions. We enforce this by adding −∞-\infty to scores for future positions before the softmax, so their weights become 0:

Python
seq = 4
mask = torch.triu(torch.ones(seq, seq), diagonal=1).bool()  # upper triangle
scores = scores.masked_fill(mask, float('-inf'))            # future → -inf

This causal mask is what makes GPT-style autoregressive generation possible.

Why attention won

RNNSelf-Attention
Path length between any two tokensO(n)O(n)O(1)O(1)
Parallelizable across sequence?NoYes
Long-range dependenciesHard (vanishing)Direct
Compute per layerO(n⋅d2)O(n \cdot d^2)O(n2⋅d)O(n^2 \cdot d)

The cost: attention is quadratic in sequence length (n2n^2 scores). For very long sequences that becomes the bottleneck — hence sparse/linear-attention research. But the constant path length and full parallelism made it the clear winner for training at scale.

Exercise

In self-attention, which vector represents 'what a token is looking for'?

Need a hint?

Query, Key, or Value?

Exercise

Why does scaled dot-product attention divide the scores by √d_k?

Need a hint?

Think about how the variance of a dot product grows with dimension.

Exercise

What is the computational complexity of self-attention in sequence length n?

Need a hint?

Every token attends to every other token.

Common mistakes

  1. Forgetting the scale factor dk\sqrt{d_k} — softmax saturates, gradients vanish, training stalls.
  2. Omitting the causal mask in a generative model — the model "cheats" by seeing the answer, then fails at inference.
  3. Confusing the dimensions: attention weights are n×nn\times n (token-to-token), the output is n×dvn\times d_v (same shape as input).

Key takeaways

  • Self-attention lets every token attend to every other in one parallel step.
  • Query·Key scores, scaled by dk\sqrt{d_k} and softmaxed, weight the Values.
  • A causal mask enables autoregressive (GPT-style) generation.
  • Cost is quadratic in sequence length — the main scaling bottleneck.

Related concepts

Deep dive
Attention Mechanisms: Self, Cross, Causal, MHA, MQA, and GQA
Includes runnable notebook

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Self-AttentionRecurrent Neural NetworksCalculus for MLThe Convolution OperationRecurrent Neural NetworksLSTM & GRUDeep Learning for Time Se…Multi-Head Attention & Po…The Transformer Architect…Inductive Bias & Equivari…The ML Project LoopScaled Dot-Product Attent…Attention Mechanisms: Sel…Dependency ParsingSession-Based Recommendat…