The RoPE of a Thousand Questions

Prelude

I have mentioned this here and there, but in 2018 I did an internship with Stanford in Russ Altman's lab for bioinformatics. They were playing around with AI models and I was assigned to try to get text properly summarized using an NLP model. The project was only 6 weeks and I got lots of help from my mentor, Stefano. After my research, I chose the LSTM due to its popularity and the numerous tutorials online, notably by Jason Brownlee.

The LSTM stands for Long Short Term Memory. It is an NLP algorithm that takes a window of text and outputs the highest probable next token. The thing is, the window was small and, as a rookie, I did not take this into account when I trained it on millions of words of text (useless compute spent).

It was not until Attention Is All You Need came out that I realized context windows are critical, and that the larger the context, the better the model performs. However, unlike LSTMs, attention models do not take inputs with sequence in mind. They take every token in the input at once and do operations on them.

Learning about positional encoding helped me understand how a token's position is taken into account despite the input being a sequence-oblivious high dimensional tensor.

AI by Hand

I was working through the AI by Hand course, the one where Professor Tom Yeh makes you compute the internals of a neural network with a pencil instead of letting a library do the hiding for you. There is something clarifying about that. You cannot hand-wave a matrix multiply when you are the one filling in the cells. The module was on Qwen, Alibaba's open model family, and somewhere between the attention scores and the decoding strategies I hit positional encoding and got stuck on a question that turned out to be the good kind of stuck.

Qwen is short for 通义千问, Tongyi Qianwen. Roughly: comprehend the meaning, answer a thousand questions. The piece I care about is 千问. 千 is qian, thousand. 问 is wen, to ask, to question. That 问 is the "wen" hiding in the English name.

A deeper look

As we stated before, attention on its own has no idea what order anything is in. Strip away the positional machinery and attention treats your sequence as an unordered set of vectors. Every token computes a query, every token advertises a key, you take dot products, you softmax, you mix the values. Nowhere in that does the index of a token enter. "Dog bites man" and "man bites dog" are, to raw attention, the same bag of three tokens. Permute the inputs and the outputs just permute right along with them. The technical word is permutation-equivariant. The plain-English version is that the mechanism is order-blind by construction.

That is a problem, because language is almost entirely about order. So we have to inject position from the outside.

However, in the world of machine learning, the tensor itself encodes the "definition" of the word relative to its relationship with other words. When you encode the index into the definition, it changes the definition, right? That was one of the questions I got answered by the session's TA responding to another classmate. Even though you are adding some positional data to the definition, that does not mean it loses its definition.

For example, if you pick up a pen, then rotate it, it is still a pen. Your brain has no trouble holding both facts: that the object is a pen, and that it is currently at a different angle. You do not get confused about what the object is after it is rotated, and the same goes for attention processing tensors with positional encoding.

Sinusoidal soil: encoding position with frequencies

The original transformer encodes position with sines and cosines at a whole spectrum of frequencies:

$PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)$

Each dimension oscillates at its own rate. Fast frequencies tick over every few tokens, slow ones drift across the entire sequence. Together they give every position a unique fingerprint, a kind of chord, and crucially a smooth one. Nearby positions get similar encodings, so the model can generalize across distances instead of memorizing each slot in isolation.

Another question that was raised in the classroom was: why not use a random vector for the positional encoding? I am not quite certain why they asked this, because the whole point of positional encoding is to have a repeatable pattern that can be used to encode the index of a token. If every token has random position data, it is like trying to draw a line on a map with a sequence of random coordinates. However, the answer was quite keen. GPT-2 used things called learned positional embeddings, which start as random vectors that gradient descent then shapes into something structured. (Gradient descent works by measuring which direction makes the loss go up the fastest, then stepping the opposite way. Picture it as feeling out the slope of a high dimensional surface and walking downhill toward the lowest point.)

The problem is that with a learned positional embedding, you can only learn positions you actually trained on. If you trained a model with a length of 4096, then 4097 has no relation to any other position, since it was never given a weight.

RoPE: stop adding, start rotating

This is the one Qwen actually uses, and it is the part where the pen stops being a metaphor.

Rotary Positional Encoding throws out the idea of adding a position vector. Instead it rotates the query and key vectors in 2D subspaces by an angle proportional to position. Take a pair of dimensions, spin them by angle $m\theta$ where $m$ is the position:

$\begin{pmatrix} x_1' \\ x_2' \end{pmatrix} = \begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}$

We were talking about disambiguating a rotated pen this whole time, and the actual mechanism turns out to be a literal rotation. I did not expect the metaphor to be the implementation.

Here is the magic property, the thing that makes RoPE more than a clever reskin. When you take the dot product of a query rotated at position $m$ against a key rotated at position $n$, the absolute angles cancel and what survives depends only on the difference $m - n$. Relative position falls out of every attention score for free, with no extra parameters and nothing to learn. The model is not told "you are at position 5, and you, key, are at position 2." It is handed "you two are three apart," every time, automatically.

And now the fruits of doing this:

• Longer context. The positional signal is multiplicative and lives inside the attention dot product, so behavior degrades gracefully as sequences grow instead of snapping. There are still limits, the rotation angles eventually wrap around and get ambiguous, but you have far more runway, and tricks like NTK-aware base scaling and YaRN stretch that runway further.
• Deeper layers. With additive encodings you inject position exactly once, at the very bottom of the network. Thirty layers later that signal has been mixed in with everything the layers wrote on top of it. RoPE re-applies position to the queries and keys at every single attention layer, because the rotation is part of the attention computation itself. Position never gets buried, because it never stops being reintroduced.

Context rot

That second point connects to something people call context rot, the way models seem to lose the thread over very long inputs. Part of it is positional. But it is also that deeper layers naturally specialize away from position and toward meaning. We know from interpretability work that middle layers do a lot of the "the capital of Kenya is" into "Nairobi" routing, write that into the residual stream, and later layers read whatever is sitting there to predict the next token. Position becoming less prominent with depth is partly a feature, not only a bug. RoPE's contribution is that position stays available on the model's terms, because the signal is strong and reintroduced at every layer, rather than being added once at the bottom and getting diluted under everything the network piles on after it.

Takeaways

The innovation in RoPE is using what we learned in pre-calculus, radians and the $2\pi$ of a circle, to encode the position of a token in an embedding with more precision. The key is that this happens at every layer of attention. In sinusoidal encoding, you apply it once at the beginning, in the first layer, and then as the layers process the stream, the positional signal gets diluted and reshaped by all the operations stacked on top of it. RoPE is reapplied at every layer, so the position signal is rebuilt fresh each time, costs almost nothing, and can preserve context much longer.

Run it

I have used Qwen and it is solid. Fast, reliable answers. I do not have much insight into the training process, but it seems that RoPE is great at connecting ideas.

I wanted to see how it compares to the old version, so I wrote a benchmark script with Claude to compare both flavors of positional encoding.

First it times the RoPE overhead across a sweep of sequence lengths, so you can see for yourself how cheap the rotation actually is relative to the attention it rides on top of.

Second, it demonstrates the relative-position property numerically. Slide a query and key pair down the sequence by the same offset and watch the score stay fixed to machine precision. Then watch plain attention fail to tell "dog bites man" from "man bites dog" at all, because permuting the keys just permutes the logits right back.

Check it out under Projects.

Inspiration

I am inspired by how one ML engineering improvement can add so much value to a model, especially one that is open source and can be studied in public. It makes me excited to see what the future holds, and the possibilities of reaping the rewards of these studies and making my own improvements in Natural Language Processing.