LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training

Sikui Zhang, Guangze Gao, Ziyun Gan +5

Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window, primarily due to the out-of-distribution (OOD) be…