Positional Encodings — Sinusoid, RoPE, ALiBi

Why attention forgets order

Plain attention computes $\text{softmax}(QK^\top / \sqrt d)\, V$. Each output token is a weighted sum over all the input tokens — but the weights depend only on the content of $Q$ and $K$, not on their indices. Permute the input sequence and every attention weight is permuted along with it. The output set is the same set.

That symmetry is fatal for any task that depends on order, which is every interesting task language has. Three families of fixes have prevailed:

• Sinusoidal (Vaswani et al., 2017): add a fixed waveform to each token embedding. Cheap, no learned parameters.
• RoPE (Su et al., 2021): rotate Q and K by an angle proportional to position. The dot product $Q_i \cdot K_j$ then depends only on the offset $j - i$.
• ALiBi (Press, Smith, Lewis, 2022): skip embeddings entirely. Add a linear penalty $-m \cdot |i - j|$ to attention scores. Brutally simple, extrapolates well.

They differ along three axes that matter at scale: parameter count, ability to extrapolate to longer sequences than seen at training, and computational cost. Each of the three sections below makes those differences visceral.

Sinusoidal — one frequency per dimension

Pick a sequence length and embedding dimension. The heatmap below shows the full positional encoding tensor: rows are positions $0, 1, \ldots, L-1$, columns are embedding dimensions $0, 1, \ldots, d-1$. Even columns are sines; odd columns are cosines. Each dimension has its own frequency, spaced geometrically from $1$ down to $1 / 10000$.

The headline property: $\mathrm{PE}_{\text{pos}+k}$ is a fixed linear function of $\mathrm{PE}_{\text{pos}}$ for any offset $k$. So a transformer can recover relative position from the absolute encoding via a learned linear map. This was the original argument for why a fixed sinusoid would work as well as a learned table.

Below: the cosine similarity between every pair of positions. Diagonal is 1 (each position with itself); the dropoff away from the diagonal tells you how distinguishable nearby positions are.

RoPE — encode position by rotating

Sinusoidal PE has a known weakness: it's added to the embedding once, and the attention layer has to spend capacity recovering relative position from the sum. RoPE inverts this: instead of adding position, it rotates the queries and keys.

Pair adjacent dimensions $(2i, 2i+1)$. For position $\text{pos}$, rotate that 2D sub-vector by an angle $\theta_i = \text{pos} \cdot 10000^{-2i/d}$. The dot product between a rotated query at position $i$ and a rotated key at position $j$ then becomes a function only of the difference $j - i$.

The headline: change positions 1 and 2 to (5, 15) or (20, 30) — the offset is 10 in both. The dot product is identical. RoPE has factored absolute position out of the attention score; only relative position remains.

ALiBi — a linear penalty on distance

The simplest of the three. Skip positional encodings at the embedding layer entirely. Add a fixed bias $-m \cdot |i - j|$ to the attention score, where the slope $m$ is hand-set per head (a geometric sequence across heads — head $h$ uses $m_h = 2^{-8h/H}$).

The matrix below shows the bias on a 32×32 attention grid for one head. Pick a slope. The bias is uniformly negative and grows with distance: it penalises attending to faraway tokens — equivalently, biases each token toward attending to its neighbours.

ALiBi's selling point: extrapolation. There's nothing in the bias matrix that depends on absolute position — only on $|i - j|$. The model trained on length 2048 can be deployed at length 16384 with no degradation in the relative-position signal it sees.

Extrapolation — train at 2k, test at 32k

The benchmark that put RoPE and ALiBi on the map: train a transformer with a max context of 2048 tokens; evaluate perplexity at 4k, 8k, 16k, 32k. A scheme that has memorised absolute positions $0$–$2047$ will fall off a cliff past 2048. A scheme that uses only relative information will degrade gracefully.

Below: the three positional-encoding schemes scored on a synthetic "relative-position recall" task. The metric is the cosine similarity between the model's representation of a query at distance $\Delta$ in training (≤ 2k) versus the same offset at test (up to 32k). High and flat is good; falling is bad. The numbers reproduce the qualitative finding from Press et al. (2022):

When to use what

The post-2022 consensus: RoPE is the modern default for autoregressive LLMs; ALiBi is preferred when extrapolation is the top concern and the model is single-task. Learned absolute is the legacy baseline; nobody picks it for a new model.

NTK-aware scaling, YaRN, and dynamic NTK are all techniques for extending RoPE-trained models beyond their training length. They rescale the base frequency to keep rotation angles in the seen range. None of them comes free — there is always some accuracy trade-off, well-documented in the long-context benchmarks (LongBench, Needle in a Haystack, RULER).

Three misconceptions

Reading list

• Vaswani et al., 2017 — Attention Is All You Need. Sinusoidal positional encoding, §3.5.
• Shaw, Uszkoreit, Vaswani, 2018 — Self-Attention with Relative Position Representations. The first relative-position scheme inside attention.
• Raffel et al., 2019 (T5) — bucketed relative-position bias added to attention scores.
• Su et al., 2021 — RoFormer: Enhanced Transformer with Rotary Position Embedding. The RoPE paper.
• Press, Smith, Lewis, 2022 — Train Short, Test Long: Attention with Linear Biases. ALiBi.
• Chen et al., 2023 — Extending Context Window of Large Language Models via Positional Interpolation. The first widely-adopted RoPE-extension trick.
• Peng et al., 2023 — YaRN: Efficient Context Window Extension of Large Language Models.
• bloc97, 2023 (LocalLlama post) — NTK-aware scaled RoPE. The community-discovered fix that made long-context RoPE practical.
• Kazemnejad et al., 2023 — The Impact of Positional Encoding on Length Generalization in Transformers. Empirical comparison of all variants.