4 citations · 5 across the 10 of their papers we have counts for
Showing 2024 · cs.DCShow all
2 papers · 2 filters
cs.DC2024
Balancing Pipeline Parallelism with Vocabulary Parallelism
Man Tsung Yeung, Penghui Qi, Min Lin +1
Pipeline parallelism is widely used to scale the training of transformer-based large language models, various works have been done to improve its throughput and memory footprint. I…
cs.DC2024★ 4 cited
Zero Bubble Pipeline Parallelism
Penghui Qi, Xinyi Wan, Guangxing Huang +1
Pipeline parallelism is one of the key components for large-scale distributed training, yet its efficiency suffers from pipeline bubbles which were deemed inevitable. In this work,…