PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

Dawei Zhu, Nan Yang, Liang Wang +4

Large Language Models (LLMs) are trained with a pre-defined context length, restricting their use in scenarios requiring long inputs. Previous efforts for adapting LLMs to a longer…