157 citations · 163 across the 6 of their papers we have counts for
Showing 2024 · cs.DCShow all
2 papers · 2 filters
cs.DC2024
Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models
Runsheng Benson Guo, Utkarsh Anand, Arthur Chen +1
Training transformer models requires substantial GPU compute and memory resources. In homogeneous clusters, distributed strategies allocate resources evenly, but this approach is i…
cs.DC2024
FreeRide: Harvesting Bubbles in Pipeline Parallelism
Jiashu Zhang, Zihan Pan, Molly +3
The occurrence of bubbles in pipeline parallelism is an inherent limitation that can account for more than 40% of the large language model (LLM) training time and is one of the mai…