Distributed Sparse SGD with Majority Voting

Abstract

Distributed learning, particularly variants of distributed stochastic gradient descent (DSGD), are widely employed to speed up training by leveraging computational resources of several workers. However, in practise, communication delay becomes a bottleneck due to the significant amount of information that needs to be exchanged between the workers and the parameter server. One of the most efficient strategies to mitigate the communication bottleneck is top-K sparsification. However, top-K sparsification requires additional communication load to represent the sparsity pattern, and the mismatch between the sparsity patterns of the workers prevents exploitation of efficient communication protocols. To address these issues, we introduce a novel majority voting based sparse communication strategy, in which the workers first seek a consensus on the structure of the sparse representation. This strategy provides a significant reduction in the communication load and allows using the same sparsity level in both communication directions. Through extensive simulations on the CIFAR-10 dataset, we show that it is possible to achieve up to x4000 compression without any loss in the test accuracy.

References in corpus (18)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Optimal Distributed Online Prediction using Mini-Batches
• Sparsified SGD with Memory
• On Analog Gradient Descent Learning over Multiple Access Fading Channels
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Dynamic Model Pruning with Feedback
• Natural Compression for Distributed Deep Learning
• Is Local SGD Better than Minibatch SGD?
• Double Quantization for Communication-Efficient Distributed Optimization
• Sparse Communication for Training Deep Networks
• rTop-k: A Statistical Estimation Approach to Distributed SGD
• Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning
• Variance Reduction with Sparse Gradients