AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

Shaowen Wang, Yuke Zheng, Tansheng Zhu +4

Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling a…