RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

Yufeng Du, Phillip Harris, Minyang Tian +5

We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis abstracts away from the specifi…