6 citations · 6 across the 2 of their papers we have counts for
3 papers
cs.DC2024★ 1 cited
LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
Diandian Gu, Peng Sun, Qinghao Hu +11
Efficiently training LLMs with long sequences is important yet challenged by the massive computation and memory requirements. Sequence parallelism has been proposed to tackle these…
cs.DC2024★ 6 cited
Characterization of Large Language Model Development in the Datacenter
Qinghao Hu, Zhisheng Ye, Zerui Wang +9
Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resour…
cs.DC2024
InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
Qiaoling Chen, Diandian Gu, Guoteng Wang +8
Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are n…