Scaled Dot-Product Attention
Scaled dot-product attention is the single operation that lets every token in a sequence directly look at every other token. This page derives the scaling factor from first principles and works through a complete 3-token example by hand.
The formula
where (queries), (keys), (values), and is sequence length.
The step-by-step procedure
- Scores: compute . Entry is the dot product of query with key : how relevant token is to token .
- Scale: divide by to prevent large magnitudes from saturating the softmax.
- Softmax: apply row-wise so each row sums to 1 (the attention weights).
- Weighted sum: multiply by : each output is a blend of all values, weighted by relevance.
Watch it run
Step through those four operations on three tokens with . The state panel shows each matrix as it is filled in, row by row; the arithmetic behind every number is worked out by hand further down this page, so you can check any step you don't believe.
Scaled dot-product attention — one pass, matrix by matrix
Three tokens, d_k = 2, with Q, K and V given directly (in a real model these are learned linear projections of the token embeddings). Every token will end up rewritten as a weighted blend of all three value vectors.
| ·0 | ·1 | |
|---|---|---|
| t1 | 2 | 0 |
| t2 | 0 | 2 |
| t3 | 2 | 2 |
| ·0 | ·1 | |
|---|---|---|
| t1 | 2 | 0 |
| t2 | 0 | 2 |
| t3 | 1 | 1 |
| ·0 | ·1 | |
|---|---|---|
| t1 | 1 | 0 |
| t2 | 0 | 1 |
| t3 | 10 | 10 |
Read each row as one token: q_i asks a question, k_j advertises what token j offers, v_j is what token j actually contributes if attended to.
The four steps of softmax(QKᵀ/√d_k)·V on three tokens with d_k = 2, in the same numbers as the hand-worked example above: score every query against every key, scale, softmax each row into a distribution, then blend the value vectors. The last step shows what the same row looks like at d_k = 64 without the scale factor — softmax saturates and the gradient vanishes.
The final step is the one worth sitting with: it re-runs row 1 at the score magnitude typical of without the scale factor, and the softmax collapses to a one-hot lookup. That saturated distribution is the failure the exists to prevent — the next section derives why.
Why divide by
Take query and key with components that are i.i.d. with mean 0 and variance 1. Their dot product is:
Each product has:
- (independence)
Summing independent unit-variance terms:
The typical score magnitude grows like . For large (e.g. ), raw scores become large, pushing softmax into its saturated region where one weight and the rest — gradient is nearly zero there, so learning stalls.
Dividing by rescales the variance back to 1:
Empirical check: for , sampling 20,000 random unit-variance pairs gives — matching exactly.
The saturation intuition
Think of softmax as a thermostat. When all scores are moderate, the distribution is spread (the model can attend to many tokens). When one score dominates by 10×, softmax assigns ~1.0 to that entry — the model becomes a hard lookup. Large without scaling causes the latter by default, which is why the scale factor is not optional.
A 3-token worked example ()
Given projected vectors directly (each row = one token):
Step 1: Scores . Entry measures how well token 's query aligns with token 's key — a higher score means "pay more attention here." Row 1: ; ; :
Token 1's query matches key 1 best (score 4), not at all with key 2 (score 0), and moderately with key 3 (score 2). After softmax, this will mean token 1 attends mostly to itself.
Step 2: Scale by . Using and :
Step 3: Softmax each row. Row 1: , , , sum :
Row 3: all three scaled scores equal ( each), so softmax is exactly uniform . Row 2 mirrors row 1: .
Every row sums to exactly 1.
Step 4: Weighted sum . Row 1:
Row 3 (uniform weights) = average of value rows: .
Token 1 attends mostly to key 1 (its strongest match), but the high value of token 3 still lifts its output. Token 3, matching all keys equally, returns the plain average of all values.
Verification in pure Python
import math
def softmax(row):
e = [math.exp(s) for s in row]
z = sum(e)
return [x / z for x in e]
Q = [[2, 0], [0, 2], [2, 2]]
K = [[2, 0], [0, 2], [1, 1]]
V = [[1, 0], [0, 1], [10, 10]]
d_k = 2
scores = [[sum(q[d]*k[d] for d in range(d_k)) for k in K] for q in Q]
scaled = [[s / math.sqrt(d_k) for s in row] for row in scores]
weights = [softmax(row) for row in scaled]
out = [[sum(weights[i][j]*V[j][d] for j in range(3)) for d in range(d_k)]
for i in range(3)]
print("scores :", scores) # [[4,0,2],[0,4,2],[4,4,4]]
print("weights :", [[round(w,3) for w in r] for r in weights])
print("output :", [[round(o,3) for o in r] for r in out])
The PyTorch one-liner
import torch, torch.nn.functional as F
Q = torch.tensor([[2.,0],[0,2],[2,2]])
K = torch.tensor([[2.,0],[0,2],[1,1]])
V = torch.tensor([[1.,0],[0,1],[10,10]])
out = F.scaled_dot_product_attention(Q.unsqueeze(0), K.unsqueeze(0), V.unsqueeze(0))
print(out.squeeze(0)) # matches the hand calculation above
Related concepts
- Self-Attention: source lesson: QKV intuition, causal masking, comparison with RNNs
- Multi-Head Attention: running multiple attention heads in parallel to capture different relationships