ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling

Yiwen Chen, Joshua Ainslie, Krzysztof Choromanski +4

Rotary Position Embedding (RoPE) has been widely adopted in transformer-based large language models. However, its log-linear frequency schedule, originally designed to produce long…