Positional Encodings: Sinusoidal, RoPE, ALiBi, and LearnedOpen in Colab

Positional Encodings: Sinusoidal, RoPE, ALiBi, and Learned

Attention is permutation-invariant: swapping token positions changes the Q and K rows in the score matrix, but the model has no explicit concept of order. Without positional encoding, "The dog bit the man" and "The man bit the dog" look identical to the attention layer — the same bag of token vectors.

Positional encodings inject order information. The methods differ critically in how well they generalize to sequence lengths longer than seen during training.

Absolute sinusoidal (original Transformer)

Vaswani et al. (2017) add a fixed, non-learned vector PEpos∈RdPE_\text{pos} \in \mathbb{R}^d to each token embedding before the first layer:

PE(pos, 2i)=sin⁡ ⁣(pos100002i/d),PE(pos, 2i+1)=cos⁡ ⁣(pos100002i/d)PE_{(\text{pos},\, 2i)} = \sin\!\left(\frac{\text{pos}}{10000^{2i/d}}\right), \qquad PE_{(\text{pos},\, 2i+1)} = \cos\!\left(\frac{\text{pos}}{10000^{2i/d}}\right)

where pos∈{0,1,…}\text{pos} \in \{0, 1, \ldots\} is the token index and i∈{0,1,…,d/2−1}i \in \{0, 1, \ldots, d/2 - 1\} indexes the dimension pair.

Frequency structure: each dimension pair (2i,2i+1)(2i, 2i+1) oscillates at a different frequency. Low-index pairs oscillate slowly (long period, long-range signal); high-index pairs oscillate quickly (short period, fine-grained local signal). The base 1000010000 sets the lowest frequency to period 2π×10000≈62,8322\pi \times 10000 \approx 62{,}832 — longer than any practical sequence.

Key property: the relative offset between positions pos\text{pos} and pos+k\text{pos}+k can be expressed as a linear function of PEposPE_\text{pos}. This means attention scores qi⋅kj\mathbf{q}_i \cdot \mathbf{k}_j can encode relative distance in principle, but in practice the model must learn this through training.

Extrapolation: the formula is defined for any position, but training only sees positions up to LtrainL_\text{train}. At longer lengths, attention patterns typically degrade because the absolute position values are out-of-distribution.

Learned absolute embeddings

Replace the fixed formula with a learned lookup table E∈RLmax×dE \in \mathbb{R}^{L_\text{max} \times d} — one embedding vector per absolute position, trained end-to-end:

inputpos=token embeddingpos+Epos\text{input}_\text{pos} = \text{token embedding}_\text{pos} + E_\text{pos}

Used in BERT, GPT-1/2. The embeddings can adapt to the training data's position distribution, often matching or slightly outperforming sinusoidal in practice on sequences within the training length.

Hard ceiling: there is no parameter for positions beyond LmaxL_\text{max}. The model cannot generalize to longer inputs without retraining or interpolation.

Relative position bias (T5 / RPE)

Instead of adding position to the token embedding, inject position as a learned bias on the attention score:

score(i,j)=qi⋅kjdk+b(i−j)\text{score}(i,j) = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}} + b(i - j)

where b(δ)b(\delta) is a scalar for relative offset δ=i−j\delta = i - j. T5 buckets the offsets logarithmically — 32 buckets cover all offsets from −128-128 to 128128, sharing parameters across similar distances.

Extrapolation: once all offsets larger than 128 saturate into the same bucket, the model applies the same bias for all distances beyond the training range — a graceful degradation rather than a catastrophic distribution shift.

Rotary Position Embeddings (RoPE)

RoPE (Su et al. 2021) encodes position by rotating query and key vectors in 2D subspaces, rather than adding a fixed offset. For dimension pair (2i,2i+1)(2i, 2i+1) at position pos\text{pos}, apply a rotation with angle θpos,i=pos/100002i/d\theta_{\text{pos},i} = \text{pos} / 10000^{2i/d}:

q~pos(2i)=qpos(2i)cos⁡θ−qpos(2i+1)sin⁡θ,q~pos(2i+1)=qpos(2i)sin⁡θ+qpos(2i+1)cos⁡θ\tilde{q}^{(2i)}_\text{pos} = q^{(2i)}_\text{pos}\cos\theta - q^{(2i+1)}_\text{pos}\sin\theta, \quad \tilde{q}^{(2i+1)}_\text{pos} = q^{(2i)}_\text{pos}\sin\theta + q^{(2i+1)}_\text{pos}\cos\theta

The crucial property: the dot product of a rotated query at position mm with a rotated key at position nn depends only on the relative offset m−nm - n:

q~m⋅k~n=f(qm,kn, m−n)\tilde{\mathbf{q}}_m \cdot \tilde{\mathbf{k}}_n = f(\mathbf{q}_m, \mathbf{k}_n,\, m - n)

So attention scores naturally encode relative position with zero extra parameters — no position embedding table to add to the residual stream.

PyTorch implementation sketch:

Python
def apply_rope(x, cos, sin):
    # x: (batch, seq, heads, d_head)
    # cos, sin: (seq, d_head/2) — precomputed for each position
    x1 = x[..., ::2]    # even dimensions
    x2 = x[..., 1::2]   # odd dimensions
    x_rotated = torch.stack([-x2, x1], dim=-1).flatten(-2)
    return x * cos.repeat_interleave(2, -1) + x_rotated * sin.repeat_interleave(2, -1)

Why RoPE is the de facto standard (2024): used in LLaMA-1/2/3, Mistral, Gemma, Falcon, PaLM-2, Qwen, and most modern open-weight LLMs. Zero parameters, exact relative-position encoding, and better extrapolation than absolute methods — especially with YaRN or NTK-aware interpolation, which extend the effective context window 4–8× without full retraining.

ALiBi (Attention with Linear Biases)

ALiBi (Press et al. 2022) adds a static, non-learned linear distance penalty to each attention score:

score(i,j)=qi⋅kjdk−mh⋅∣i−j∣\text{score}(i,j) = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}} - m_h \cdot |i - j|

where mh=2−8h/Hm_h = 2^{-8h/H} is a head-specific slope (different heads apply different strengths of recency bias).

Extrapolation: because the penalty is a fixed, monotonically increasing function of distance, it naturally handles sequences longer than training. Far tokens receive larger penalties without any distribution shift. Models with ALiBi are trained on shorter sequences and deployed at longer ones out of the box — MPT and BLOOM use this approach.

No positional parameters at all. The downside: the built-in recency bias may be suboptimal for tasks requiring genuine long-range attention (e.g., code completion referencing a function defined thousands of tokens earlier).

Comparison

MethodExtra paramsRelative position?Length extrapolationUsed in
Sinusoidal0ImplicitPoor beyond LtrainL_\text{train}Original Transformer
Learned absoluteLmax×dL_\text{max} \times dNoNone past LmaxL_\text{max}BERT, GPT-2
T5 relative bias~32 per headYesGraceful (bucket saturation)T5, DeBERTa
RoPE0Yes (exact)Moderate; 4–8× with YaRNLLaMA, Mistral, Gemma
ALiBi0Yes (linear penalty)ExcellentMPT, BLOOM

Length extrapolation ≠ long-context capability

Extrapolating to longer positions is only useful if the model can actually retrieve information from those positions. Without architectural changes (sliding window, RoPE scaling), models often fail to attend to relevant content in the middle of a very long sequence (the "lost in the middle" phenomenon) regardless of which position encoding is used.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Positional Encodings: Sinusoidal,…Attention Mechanisms: Sel…Multi-Head Attention & Po…Modern Attention Mechanis…