Spiral RoPE: Rotate Your Rotary Positional Embeddings in the 2D Plane

Haoyu Liu, Sucheng Ren, Tingyu Zhu +5

Rotary Position Embedding (RoPE) is the de facto positional encoding in large language models due to its ability to encode relative positions and support length extrapolation. When…