90 citations · 92 across the 5 of their papers we have counts for
5 papers
Don't Let a Few Network Failures Slow the Entire AllReduce
Peiqing Chen, Jiedong Jiang, Nengneng Yu +4
Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication librarie…
Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
Xinglin Pan, Shaohuai Shi, Wenxiang Lin +4
The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert act…
DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining
Lin Zhang, Shaohuai Shi, Xiaowen Chu +3
Communication scheduling has been shown to be effective in accelerating distributed training, which enables all-reduce communications to be overlapped with backpropagation computat…
A Quantitative Survey of Communication Optimizations in Distributed Deep Learning
Shaohuai Shi, Zhenheng Tang, Xiaowen Chu +3
Nowadays, large and complex deep learning (DL) models are increasingly trained in a distributed manner across multiple worker machines, in which extensive communications between wo…
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
Zhenheng Tang, Shaohuai Shi, Wei Wang +2
Distributed deep learning (DL) has become prevalent in recent years to reduce training time by leveraging multiple computing devices (e.g., GPUs/TPUs) due to larger models and data…