3 papers
cs.DC2026
Revisiting Parameter Server in LLM Post-Training
Xinyi Wan, Penghui Qi, Guangxing Huang +3
Modern data parallel (DP) training favors collective communication over parameter servers (PS) for its simplicity and efficiency under balanced workloads. However, the balanced wor…
cs.LG2025
PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Xinyi Wan, Penghui Qi, Guangxing Huang +2
Pipeline parallelism (PP) is widely used for training large language models (LLMs), yet its scalability is often constrained by high activation memory consumption as the number of…
cs.DC2024
Balancing Pipeline Parallelism with Vocabulary Parallelism
Man Tsung Yeung, Penghui Qi, Min Lin +1
Pipeline parallelism is widely used to scale the training of transformer-based large language models, various works have been done to improve its throughput and memory footprint. I…