Extending Context Window of Large Language Models from a Distributional Perspective

Yingsheng Wu, Yuxuan Gu, Xiaocheng Feng +5

Scaling the rotary position embedding (RoPE) has become a common method for extending the context window of RoPE-based large language models (LLMs). However, existing scaling metho…