Positional Encoding via Token-Aware Phase Attention

Yu Wang, Sheng Shen, Rémi Munos +2

We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model…