163 citations · 201 across the 5 of their papers we have counts for
4 papers · 1 filter
Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
Guanhua Wang, Chengming Zhang, Zheyu Shen +2
Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs for parallelizing and accelerating the training process. Communicati…
FastPersist: Accelerating Model Checkpointing in Deep Learning
Guanhua Wang, Olatunji Ruwase, Bing Xie +1
Model checkpoints are critical Deep Learning (DL) artifacts that enable fault tolerance for training and downstream applications, such as inference. However, writing checkpoints to…
ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
Guanhua Wang, Heyang Qin, Sam Ade Jacobs +6
Zero Redundancy Optimizer (ZeRO) has been used to train a wide range of large language models on massive GPUs clusters due to its ease of use, efficiency, and good scalability. How…
Blink: Fast and Generic Collectives for Distributed ML
Guanhua Wang, Shivaram Venkataraman, Amar Phanishayee +3
Model parameter synchronization across GPUs introduces high overheads for data-parallel training at scale. Existing parameter synchronization protocols cannot effectively leverage…