28 citations · 38 across the 15 of their papers we have counts for
Showing cs.DCShow all
3 papers · 1 filter
cs.DC2026
Revisiting Parameter Server in LLM Post-Training
Xinyi Wan, Penghui Qi, Guangxing Huang +3
Modern data parallel (DP) training favors collective communication over parameter servers (PS) for its simplicity and efficiency under balanced workloads. However, the balanced wor…
cs.DC2024
Balancing Pipeline Parallelism with Vocabulary Parallelism
Man Tsung Yeung, Penghui Qi, Min Lin +1
Pipeline parallelism is widely used to scale the training of transformer-based large language models, various works have been done to improve its throughput and memory footprint. I…
cs.DC2024★ 4 cited
Zero Bubble Pipeline Parallelism
Penghui Qi, Xinyi Wan, Guangxing Huang +1
Pipeline parallelism is one of the key components for large-scale distributed training, yet its efficiency suffers from pipeline bubbles which were deemed inevitable. In this work,…