Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models

Junfeng Tian, Da Zheng, Yang Cheng +3

Large language models (LLM) have prioritized expanding the context window from which models can incorporate more information. However, training models to handle long contexts prese…