4 citations · 5 across the 11 of their papers we have counts for
Showing 2025 · cs.DCShow all
2 papers · 2 filters
cs.DC2025
BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
Ao Sun, Weilin Zhao, Xu Han +4
Existing methods for training LLMs on long-sequence data, such as Tensor Parallelism and Context Parallelism, exhibit low Model FLOPs Utilization as sequence lengths and number of…
cs.DC2025
Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training
Shiju Wang, Yujie Wang, Ao Sun +6
Long context training is crucial for extending LLM context windows. Existing schemes, such as sequence parallelism, incur substantial communication overhead. Pipeline parallelism (…