Rethinking gradient sparsification as total error minimization

Atal Narayan Sahu, Aritra Dutta, Ahmed M. Abdelmoniem +3

Gradient compression is a widely-established remedy to tackle the communication bottleneck in distributed training of large deep neural networks (DNNs). Under the error-feedback fr…