138 citations · 216 across the 16 of their papers we have counts for
Showing cs.DCShow all
2 papers · 1 filter
cs.DC2026
LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
Xinyi Liu, Yujie Wang, Fangcheng Fu +4
Expert parallelism is vital for effectively training Mixture-of-Experts (MoE) models, enabling different devices to host distinct experts, with each device processing different inp…
cs.DC2024★ 2 cited
FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
Yujie Wang, Shiju Wang, Shenhan Zhu +7
Extending the context length (i.e., the maximum supported sequence length) of LLMs is of paramount significance. To facilitate long context training of LLMs, sequence parallelism h…