Sparse Communication for Training Deep Networks
arXiv:2009.09271
Abstract
Synchronous stochastic gradient descent (SGD) is the most common method used for distributed training of deep learning models. In this algorithm, each worker shares its local gradients with others and updates the parameters using the average gradients of all workers. Although distributed training reduces the computation time, the communication overhead associated with the gradient exchange forms a scalability bottleneck for the algorithm. There are many compression techniques proposed to reduce the number of gradients that needs to be communicated. However, compressing the gradients introduces yet another overhead to the problem. In this work, we study several compression schemes and identify how three key parameters affect the performance. We also provide a set of insights on how to increase performance and introduce a simple sparsification scheme, random-block sparsification, that reduces communication while keeping the performance close to standard SGD.
References in corpus (11)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Sparsified SGD with Memory
- Don't Use Large Mini-Batches, Use Local SGD
- Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
- Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
- PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
- Quantized Frank-Wolfe: Faster Optimization, Lower Communication, and Projection Free
- SparCML: High-Performance Sparse Communication for Machine Learning