How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

Xinyi Wu, Siyuan Liu, Ali Jadbabaie

Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what det…