Scaled Dot-Product AttentionOpen in Colab

Scaled Dot-Product Attention

Scaled dot-product attention is the single operation that lets every token in a sequence directly look at every other token. This page derives the 1/dk1/\sqrt{d_k} scaling factor from first principles and works through a complete 3-token example by hand.

The formula

Attention(Q,K,V)=softmax ⁣(QK⊤dk)V\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

where Q∈Rn×dkQ \in \mathbb{R}^{n \times d_k} (queries), K∈Rn×dkK \in \mathbb{R}^{n \times d_k} (keys), V∈Rn×dvV \in \mathbb{R}^{n \times d_v} (values), and nn is sequence length.

The step-by-step procedure

  1. Scores: compute QK⊤QK^\top. Entry (i,j)(i,j) is the dot product of query ii with key jj: how relevant token jj is to token ii.
  2. Scale: divide by dk\sqrt{d_k} to prevent large magnitudes from saturating the softmax.
  3. Softmax: apply row-wise so each row sums to 1 (the attention weights).
  4. Weighted sum: multiply by VV: each output is a blend of all values, weighted by relevance.

Watch it run

Step through those four operations on three tokens with dk=2d_k = 2. The state panel shows each matrix as it is filled in, row by row; the arithmetic behind every number is worked out by hand further down this page, so you can check any step you don't believe.

Scaled dot-product attention — one pass, matrix by matrix

1/14

Three tokens, d_k = 2, with Q, K and V given directly (in a real model these are learned linear projections of the token embeddings). Every token will end up rewritten as a weighted blend of all three value vectors.

python
1def softmax(row):
2 e = [exp(s) for s in row]
3 return [x / sum(e) for x in e]
4
5Q = [[2, 0], [0, 2], [2, 2]] # one row per token
6K = [[2, 0], [0, 2], [1, 1]]
7V = [[1, 0], [0, 1], [10, 10]]
8d_k = 2
9
10# (i, j): how relevant is key j to query i?
11scores = [[dot(q, k) for k in K] for q in Q]
12scaled = [[s / sqrt(d_k) for s in row]
13 for row in scores]
14# each row becomes a distribution summing to 1
15weights = [softmax(row) for row in scaled]
16out = [[sum(weights[i][j] * V[j][d]
17 for j in range(n))
18 for d in range(d_v)]
19 for i in range(n)]
state
Q (queries)
·0·1
t120
t202
t322
K (keys)
·0·1
t120
t202
t311
V (values)
·0·1
t110
t201
t31010

Read each row as one token: q_i asks a question, k_j advertises what token j offers, v_j is what token j actually contributes if attended to.

The four steps of softmax(QKᵀ/√d_k)·V on three tokens with d_k = 2, in the same numbers as the hand-worked example above: score every query against every key, scale, softmax each row into a distribution, then blend the value vectors. The last step shows what the same row looks like at d_k = 64 without the scale factor — softmax saturates and the gradient vanishes.

The final step is the one worth sitting with: it re-runs row 1 at the score magnitude typical of dk=64d_k = 64 without the scale factor, and the softmax collapses to a one-hot lookup. That saturated distribution is the failure the 1/dk1/\sqrt{d_k} exists to prevent — the next section derives why.

Why divide by dk\sqrt{d_k}

Take query q\mathbf{q} and key k\mathbf{k} with components that are i.i.d. with mean 0 and variance 1. Their dot product is:

q⋅k=∑i=1dkqiki\mathbf{q}\cdot\mathbf{k} = \sum_{i=1}^{d_k} q_i k_i

Each product qikiq_i k_i has:

  • E[qiki]=E[qi] E[ki]=0\mathbb{E}[q_i k_i] = \mathbb{E}[q_i]\,\mathbb{E}[k_i] = 0 (independence)
  • Var⁡(qiki)=E[qi2] E[ki2]=1⋅1=1\operatorname{Var}(q_i k_i) = \mathbb{E}[q_i^2]\,\mathbb{E}[k_i^2] = 1 \cdot 1 = 1

Summing dkd_k independent unit-variance terms:

Var⁡(q⋅k)=∑i=1dk1=dk\operatorname{Var}(\mathbf{q}\cdot\mathbf{k}) = \sum_{i=1}^{d_k} 1 = d_k

The typical score magnitude grows like dk\sqrt{d_k}. For large dkd_k (e.g. dk=64d_k = 64), raw scores become large, pushing softmax into its saturated region where one weight ≈1\approx 1 and the rest ≈0\approx 0 — gradient is nearly zero there, so learning stalls.

Dividing by dk\sqrt{d_k} rescales the variance back to 1:

Var⁡ ⁣(q⋅kdk)=dkdk=1\operatorname{Var}\!\left(\frac{\mathbf{q}\cdot\mathbf{k}}{\sqrt{d_k}}\right) = \frac{d_k}{d_k} = 1

Empirical check: for dk∈{2,8,64}d_k \in \{2, 8, 64\}, sampling 20,000 random unit-variance q,k\mathbf{q},\mathbf{k} pairs gives Var⁡≈2.0, 8.0, 64\operatorname{Var} \approx 2.0,\,8.0,\,64 — matching dkd_k exactly.

The saturation intuition

Think of softmax as a thermostat. When all scores are moderate, the distribution is spread (the model can attend to many tokens). When one score dominates by 10×, softmax assigns ~1.0 to that entry — the model becomes a hard lookup. Large dkd_k without scaling causes the latter by default, which is why the scale factor is not optional.

A 3-token worked example (dk=2d_k = 2)

Given projected vectors directly (each row = one token):

Q=[200222],K=[200211],V=[10011010]Q = \begin{bmatrix}2 & 0\\0 & 2\\2 & 2\end{bmatrix}, \quad K = \begin{bmatrix}2 & 0\\0 & 2\\1 & 1\end{bmatrix}, \quad V = \begin{bmatrix}1 & 0\\0 & 1\\10 & 10\end{bmatrix}

Step 1: Scores QK⊤QK^\top. Entry (i,j)=qi⋅kj(i,j) = \mathbf{q}_i \cdot \mathbf{k}_j measures how well token ii's query aligns with token jj's key — a higher score means "pay more attention here." Row 1: [2,0]⋅[2,0]=4[2,0]\cdot[2,0]=4; [2,0]⋅[0,2]=0[2,0]\cdot[0,2]=0; [2,0]⋅[1,1]=2[2,0]\cdot[1,1]=2:

QK⊤=[402042444]QK^\top = \begin{bmatrix}4 & 0 & 2\\0 & 4 & 2\\4 & 4 & 4\end{bmatrix}

Token 1's query matches key 1 best (score 4), not at all with key 2 (score 0), and moderately with key 3 (score 2). After softmax, this will mean token 1 attends mostly to itself.

Step 2: Scale by 1/2≈0.7071/\sqrt{2} \approx 0.707. Using 4/2=22≈2.8284/\sqrt{2} = 2\sqrt{2} \approx 2.828 and 2/2=2≈1.4142/\sqrt{2} = \sqrt{2} \approx 1.414:

QK⊤2=[2.82801.41402.8281.4142.8282.8282.828]\frac{QK^\top}{\sqrt{2}} = \begin{bmatrix}2.828 & 0 & 1.414\\0 & 2.828 & 1.414\\2.828 & 2.828 & 2.828\end{bmatrix}

Step 3: Softmax each row. Row 1: e2.828=16.92e^{2.828}=16.92, e0=1.00e^0=1.00, e1.414=4.11e^{1.414}=4.11, sum =22.03= 22.03:

softmax(row 1)=122.03[16.92, 1.00, 4.11]=[0.768, 0.045, 0.187]\text{softmax}(\text{row 1}) = \frac{1}{22.03}[16.92,\,1.00,\,4.11] = [0.768,\,0.045,\,0.187]

Row 3: all three scaled scores equal (2.8282.828 each), so softmax is exactly uniform [13,13,13][\frac{1}{3},\frac{1}{3},\frac{1}{3}]. Row 2 mirrors row 1: [0.045, 0.768, 0.187][0.045,\,0.768,\,0.187].

weights=[0.7680.0450.1870.0450.7680.1870.3330.3330.333]\text{weights} = \begin{bmatrix}0.768 & 0.045 & 0.187\\0.045 & 0.768 & 0.187\\0.333 & 0.333 & 0.333\end{bmatrix}

Every row sums to exactly 1.

Step 4: Weighted sum weights⋅V\text{weights}\cdot V. Row 1:

0.768[1,0]+0.045[0,1]+0.187[10,10]=[0.768+1.870,  0.045+1.870]=[2.638,  1.915]0.768[1,0] + 0.045[0,1] + 0.187[10,10] = [0.768 + 1.870,\; 0.045 + 1.870] = [2.638,\;1.915]

Row 3 (uniform weights) = average of value rows: 13([1,0]+[0,1]+[10,10])=[3.667, 3.667]\frac{1}{3}([1,0]+[0,1]+[10,10]) = [3.667,\,3.667].

output=[2.6381.9151.9152.6383.6673.667]\text{output} = \begin{bmatrix}2.638 & 1.915\\1.915 & 2.638\\3.667 & 3.667\end{bmatrix}

Token 1 attends mostly to key 1 (its strongest match), but the high value of token 3 still lifts its output. Token 3, matching all keys equally, returns the plain average of all values.

Verification in pure Python

Python
import math

def softmax(row):
    e = [math.exp(s) for s in row]
    z = sum(e)
    return [x / z for x in e]

Q = [[2, 0], [0, 2], [2, 2]]
K = [[2, 0], [0, 2], [1, 1]]
V = [[1, 0], [0, 1], [10, 10]]
d_k = 2

scores  = [[sum(q[d]*k[d] for d in range(d_k)) for k in K] for q in Q]
scaled  = [[s / math.sqrt(d_k) for s in row] for row in scores]
weights = [softmax(row) for row in scaled]
out = [[sum(weights[i][j]*V[j][d] for j in range(3)) for d in range(d_k)]
       for i in range(3)]

print("scores  :", scores)                               # [[4,0,2],[0,4,2],[4,4,4]]
print("weights :", [[round(w,3) for w in r] for r in weights])
print("output  :", [[round(o,3) for o in r] for r in out])

The PyTorch one-liner

Python
import torch, torch.nn.functional as F

Q = torch.tensor([[2.,0],[0,2],[2,2]])
K = torch.tensor([[2.,0],[0,2],[1,1]])
V = torch.tensor([[1.,0],[0,1],[10,10]])

out = F.scaled_dot_product_attention(Q.unsqueeze(0), K.unsqueeze(0), V.unsqueeze(0))
print(out.squeeze(0))  # matches the hand calculation above

Related concepts

  • Self-Attention: source lesson: QKV intuition, causal masking, comparison with RNNs
  • Multi-Head Attention: running multiple attention heads in parallel to capture different relationships

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Scaled Dot-Product AttentionAttention Mechanisms: Sel…Self-AttentionMulti-Head Attention & Po…