You could have designed state of the art positional encoding
TL;DR
The Hugging Face blog post explains how to iteratively improve positional encoding for transformer self‑attention, culminating in Rotary Positional Encoding (RoPE) used in Llama 3.2 and other modern models, and why adding positional information is essential for distinguishing tokens that appear in different positions.
Problem Statement
Self‑attention in transformers is permutation equivariant; without positional information the model cannot tell whether two identical tokens refer to different entities.
Motivating Example
In the sentence “The dog chased another dog”, the two occurrences of “dog” produce identical self‑attention outputs when no positional encoding is added, proving that positional information is needed to differentiate them.
Desirable Properties
An ideal positional encoding should: (1) give each position a unique, length‑independent code; (2) allow a simple linear computation to obtain the encoding for position p + k from the encoding for p; (3) generalize to sequence lengths longer than those seen during training; (4) be generated by a deterministic process the model can learn; and (5) extend naturally to multiple dimensions for modalities such as images.
Integer Position Encoding
Adding the raw integer position to each embedding component creates values that far exceed the typical embedding magnitude, harming the signal‑to‑noise ratio and making learning difficult.
Binary Position Encoding
Representing the position as a binary string and spreading its bits across the embedding dimension yields values in [0,1] that are consistent across sequence lengths, but the resulting pattern is discrete and “jumpy”, which is not ideal for gradient‑based optimization.
Sinusoidal positional encoding
Using sinusoids and cosinusoids with geometrically increasing wavelengths produces smooth, continuous encodings. The encoding for position p and dimension 2i is sin(p / 10000^{2i/d}) and for 2i+1 is cos(p / 10000^{2i/d}), where d is the model dimension. This scheme satisfies the uniqueness and linear‑relation properties because a fixed offset k corresponds to a rotation matrix acting on each (sin,cos) pair.
Absolute vs Relative Position Encoding
Absolute position tells where a token occurs in the whole sequence, but meaning usually depends on how a token relates to nearby tokens. Sinusoidal encoding can be interpreted as encoding relative position via a rotation, which motivates a shift from additive to multiplicative combination with the query and key vectors.
Positional encoding in context
In self‑attention the dot product QKᵀ determines how much each key influences a query. Because the dot product equals ‖Q‖‖K‖cos θ, rotating Q or K changes the angle θ and thus the attention weight without altering the vector norms, preserving the semantic information stored in the norm.
Rotary Positional Encoding (RoPE)
RoPE applies the rotation matrix derived from the sinusoidal scheme directly to pairs of dimensions in the query and key vectors before their dot product. For each pair i the rotation angle is ω_i · p with ω_i = 1 / 10000^{2i/d}. The operation can be performed element‑wise as: - q′{2i} = q{2i} · cos(p θ_i) − q_{2i+1} · sin(p θ_i) - q′{2i+1}= q{2i} · sin(p θ_i) + q_{2i+1} · cos(p θ_i) (and similarly for k). This multiplicative approach encodes relative position while keeping the vector norm unchanged.
Extending RoPE to n‑Dimensions
For 2‑D data such as images, RoPE treats each spatial dimension independently: it rotates pairs within the x‑dimension to encode horizontal offset and pairs within the y‑dimension to encode vertical offset. This avoids mixing x and y information and generalizes to any number of dimensions by allocating disjoint sub‑spaces of the embedding to each dimension.
The future of positional encoding
Recent analysis shows that models tend to rely on the lowest frequencies of RoPE, and removing (rather than rotating) those frequencies can improve performance on some models. Future work may draw from signal‑processing tools such as wavelets or hierarchical schemes, and seek encodings that stay robust under low‑precision quantization.
Conclusion
Positional encoding should not be an afterthought; it directly addresses the permutation‑equivariance limitation of self‑attention. By iteratively refining the encoding—from raw integers to binary, to sinusoids, and finally to a multiplicative rotation (RoPE)—we arrive at a scheme that satisfies the desired properties and is widely used in state‑of‑the‑art transformers.