SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks

Linnan Wang, Wei Wu, Junyu Zhang +4

The performance and efficiency of distributed training of Deep Neural Networks highly depend on the performance of gradient averaging among all participating nodes, which is bounde…