28 citations · 38 across the 15 of their papers we have counts for
Showing 2024Show all
3 papers · 1 filter
cs.DC2024
Balancing Pipeline Parallelism with Vocabulary Parallelism
Man Tsung Yeung, Penghui Qi, Min Lin +1
Pipeline parallelism is widely used to scale the training of transformer-based large language models, various works have been done to improve its throughput and memory footprint. I…
cs.LG2024
Pipeline Parallelism with Controllable Memory
Penghui Qi, Xinyi Wan, Nyamdavaa Amar +1
Pipeline parallelism has been widely explored, but most existing schedules lack a systematic methodology. In this paper, we propose a framework to decompose pipeline schedules as r…
cs.DC2024★ 4 cited
Zero Bubble Pipeline Parallelism
Penghui Qi, Xinyi Wan, Guangxing Huang +1
Pipeline parallelism is one of the key components for large-scale distributed training, yet its efficiency suffers from pipeline bubbles which were deemed inevitable. In this work,…