Efficient Scaling of LLM Training with Flexible Context Parallelism

Yifan Niu, Han Xiao, Dongyi Liu +2

Scaling long-context capabilities is crucial for Large Language Models (LLMs). However, real-world data contain a large number of sequences with heterogeneous lengths. Existing tra…