Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees

Shaohuai Shi, Zhenheng Tang, Qiang Wang +2

To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. Ho…