Periodic RoPE for Infinite Context LLMs

Simin Huo

The ability to process ultra-long contexts is crucial for large language models (LLMs) to perform long-horizon tasks. While recent efforts have extended context windows to 1M and b…