306 citations · 764 across the 49 of their papers we have counts for
6 papers · 1 filter
MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning
Shaohuai Shi, Xiaowen Chu, Bo Li
Distributed synchronous stochastic gradient descent has been widely used to train deep neural networks (DNNs) on computer clusters. With the increase of computational power, networ…
Understanding Top-k Sparsification in Distributed Deep Learning
Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung +1
Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes th…
Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models
Xin He, Shihao Wang, Shaohuai Shi +10
Skin disease is one of the most common types of human diseases, which may happen to everyone regardless of age, gender or race. Due to the high visual diversity, human diagnosis hi…
Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees
Shaohuai Shi, Zhenheng Tang, Qiang Wang +2
To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. Ho…
Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
Yuxin Wang, Qiang Wang, Shaohuai Shi +4
Deep learning has become widely used in complex AI applications. Yet, training a deep neural network (DNNs) model requires a considerable amount of calculations, long running time,…
A Distributed Synchronous SGD Algorithm with Global Top- Sparsification for Low Bandwidth Networks
Shaohuai Shi, Qiang Wang, Kaiyong Zhao +4
Distributed synchronous stochastic gradient descent (S-SGD) has been widely used in training large-scale deep neural networks (DNNs), but it typically requires very high communicat…