Positional Encodings: Sinusoidal, RoPE, ALiBi, and Learned
Attention is permutation-invariant: swapping token positions changes the Q and K rows in the score matrix, but the model has no explicit concept of order. Without positional encoding, "The dog bit the man" and "The man bit the dog" look identical to the attention layer — the same bag of token vectors.
Positional encodings inject order information. The methods differ critically in how well they generalize to sequence lengths longer than seen during training.
Absolute sinusoidal (original Transformer)
Vaswani et al. (2017) add a fixed, non-learned vector to each token embedding before the first layer:
where is the token index and indexes the dimension pair.
Frequency structure: each dimension pair oscillates at a different frequency. Low-index pairs oscillate slowly (long period, long-range signal); high-index pairs oscillate quickly (short period, fine-grained local signal). The base sets the lowest frequency to period — longer than any practical sequence.
Key property: the relative offset between positions and can be expressed as a linear function of . This means attention scores can encode relative distance in principle, but in practice the model must learn this through training.
Extrapolation: the formula is defined for any position, but training only sees positions up to . At longer lengths, attention patterns typically degrade because the absolute position values are out-of-distribution.
Learned absolute embeddings
Replace the fixed formula with a learned lookup table — one embedding vector per absolute position, trained end-to-end:
Used in BERT, GPT-1/2. The embeddings can adapt to the training data's position distribution, often matching or slightly outperforming sinusoidal in practice on sequences within the training length.
Hard ceiling: there is no parameter for positions beyond . The model cannot generalize to longer inputs without retraining or interpolation.
Relative position bias (T5 / RPE)
Instead of adding position to the token embedding, inject position as a learned bias on the attention score:
where is a scalar for relative offset . T5 buckets the offsets logarithmically — 32 buckets cover all offsets from to , sharing parameters across similar distances.
Extrapolation: once all offsets larger than 128 saturate into the same bucket, the model applies the same bias for all distances beyond the training range — a graceful degradation rather than a catastrophic distribution shift.
Rotary Position Embeddings (RoPE)
RoPE (Su et al. 2021) encodes position by rotating query and key vectors in 2D subspaces, rather than adding a fixed offset. For dimension pair at position , apply a rotation with angle :
The crucial property: the dot product of a rotated query at position with a rotated key at position depends only on the relative offset :
So attention scores naturally encode relative position with zero extra parameters — no position embedding table to add to the residual stream.
PyTorch implementation sketch:
def apply_rope(x, cos, sin):
# x: (batch, seq, heads, d_head)
# cos, sin: (seq, d_head/2) — precomputed for each position
x1 = x[..., ::2] # even dimensions
x2 = x[..., 1::2] # odd dimensions
x_rotated = torch.stack([-x2, x1], dim=-1).flatten(-2)
return x * cos.repeat_interleave(2, -1) + x_rotated * sin.repeat_interleave(2, -1)
Why RoPE is the de facto standard (2024): used in LLaMA-1/2/3, Mistral, Gemma, Falcon, PaLM-2, Qwen, and most modern open-weight LLMs. Zero parameters, exact relative-position encoding, and better extrapolation than absolute methods — especially with YaRN or NTK-aware interpolation, which extend the effective context window 4–8× without full retraining.
ALiBi (Attention with Linear Biases)
ALiBi (Press et al. 2022) adds a static, non-learned linear distance penalty to each attention score:
where is a head-specific slope (different heads apply different strengths of recency bias).
Extrapolation: because the penalty is a fixed, monotonically increasing function of distance, it naturally handles sequences longer than training. Far tokens receive larger penalties without any distribution shift. Models with ALiBi are trained on shorter sequences and deployed at longer ones out of the box — MPT and BLOOM use this approach.
No positional parameters at all. The downside: the built-in recency bias may be suboptimal for tasks requiring genuine long-range attention (e.g., code completion referencing a function defined thousands of tokens earlier).
Comparison
| Method | Extra params | Relative position? | Length extrapolation | Used in |
|---|---|---|---|---|
| Sinusoidal | 0 | Implicit | Poor beyond | Original Transformer |
| Learned absolute | No | None past | BERT, GPT-2 | |
| T5 relative bias | ~32 per head | Yes | Graceful (bucket saturation) | T5, DeBERTa |
| RoPE | 0 | Yes (exact) | Moderate; 4–8× with YaRN | LLaMA, Mistral, Gemma |
| ALiBi | 0 | Yes (linear penalty) | Excellent | MPT, BLOOM |
Length extrapolation ≠ long-context capability
Extrapolating to longer positions is only useful if the model can actually retrieve information from those positions. Without architectural changes (sliding window, RoPE scaling), models often fail to attend to relevant content in the middle of a very long sequence (the "lost in the middle" phenomenon) regardless of which position encoding is used.
Related concepts
- Attention Mechanisms: self-attention, cross-attention, MHA, MQA, GQA
- Multi-Head Attention & Positional Encoding: lesson covering MHA and sinusoidal encoding
- Modern Attention Mechanisms: RoPE and FlashAttention in production LLMs