306 citations · 764 across the 49 of their papers we have counts for
7 papers · 1 filter
Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
Shaohuai Shi, Xianhao Zhou, Shutao Song +21
Distributed training techniques have been widely deployed in large-scale deep neural networks (DNNs) training on dense-GPU clusters. However, on public cloud clusters, due to the m…
Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format
Shaohuai Shi, Qiang Wang, Xiaowen Chu
Multiplication of a sparse matrix to a dense matrix (SpDM) is widely used in many areas like scientific computing and machine learning. However, existing works under-look the perfo…
A Quantitative Survey of Communication Optimizations in Distributed Deep Learning
Shaohuai Shi, Zhenheng Tang, Xiaowen Chu +3
Nowadays, large and complex deep learning (DL) models are increasingly trained in a distributed manner across multiple worker machines, in which extensive communications between wo…
FADNet: A Fast and Accurate Network for Disparity Estimation
Qiang Wang, Shaohuai Shi, Shizhen Zheng +2
Deep neural networks (DNNs) have achieved great success in the area of computer vision. The disparity estimation problem tends to be addressed by DNNs which achieve much better pre…
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
Zhenheng Tang, Shaohuai Shi, Wei Wang +2
Distributed deep learning (DL) has become prevalent in recent years to reduce training time by leveraging multiple computing devices (e.g., GPUs/TPUs) due to larger models and data…
Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
Qiang Wang, Shaohuai Shi, Canhui Wang +1
Distributed Deep Learning (DDL) has rapidly grown its popularity since it helps boost the training performance on high-performance GPU clusters. Efficient job scheduling is indispe…
Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection
Zhenheng Tang, Shaohuai Shi, Xiaowen Chu
Distributed learning techniques such as federated learning have enabled multiple workers to train machine learning models together to reduce the overall training time. However, cur…