Showing cs.DCShow all
3 papers · 1 filter
cs.DC2026
StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence
Wenxuan Zhao, Yingfa Chen, Xu Han +7
Long-context training is increasingly important for large language models, and linear attention and state space models have become popular for improving long-context efficiency. Ho…
cs.DC2025
BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
Ao Sun, Weilin Zhao, Xu Han +4
Existing methods for training LLMs on long-sequence data, such as Tensor Parallelism and Context Parallelism, exhibit low Model FLOPs Utilization as sequence lengths and number of…
cs.DC2025
Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training
Shiju Wang, Yujie Wang, Ao Sun +6
Long context training is crucial for extending LLM context windows. Existing schemes, such as sequence parallelism, incur substantial communication overhead. Pipeline parallelism (…