4 papers
StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence
Wenxuan Zhao, Yingfa Chen, Xu Han +7
Long-context training is increasingly important for large language models, and linear attention and state space models have become popular for improving long-context efficiency. Ho…
InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
Shiju Wang, Yujie Wang, Ao Sun +5
Long context training is crucial for LLM's context extension. Existing schemes, such as sequence parallelism, incur substantial communication overhead. Pipeline parallelism (PP) re…
CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Sampling
Zekai Qu, Yinxu Pan, Ao Sun +2
Reinforcement learning (RL) post-training has become a trending paradigm for enhancing the capabilities of large language models (LLMs). Most existing RL systems for LLMs operate i…
BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
Ao Sun, Weilin Zhao, Xu Han +4
Existing methods for training LLMs on long-sequence data, such as Tensor Parallelism and Context Parallelism, exhibit low Model FLOPs Utilization as sequence lengths and number of…