Understanding Top-k Sparsification in Distributed Deep Learning

Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung +1

Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes th…