MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning

Shaohuai Shi, Xiaowen Chu, Bo Li

Distributed synchronous stochastic gradient descent has been widely used to train deep neural networks (DNNs) on computer clusters. With the increase of computational power, networ…