NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
arXiv:1908.06077
Abstract
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed to perform parallel model training. One popular communication-compression method for data-parallel SGD is QSGD (Alistarh et al., 2017), which quantizes and encodes gradients to reduce communication costs. The baseline variant of QSGD provides strong theoretical guarantees, however, for practical purposes, the authors proposed a heuristic variant which we call QSGDinf, which demonstrated impressive empirical gains for distributed training of large neural networks. In this paper, we build on this work to propose a new gradient quantization scheme, and show that it has both stronger theoretical guarantees than QSGD, and matches and exceeds the empirical performance of the QSGDinf heuristic and of other compression methods.
arXiv comment42 pages, 21 figures. To appear in the Journal of Machine Learning Research (JMLR)
References in corpus (16)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- Binarized Neural Networks
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Horovod: fast and easy distributed deep learning in TensorFlow
- Convolutional Neural Networks using Logarithmic Data Representation
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
- ATOMO: Communication-efficient Learning via Atomic Sparsification
- Communication Compression for Decentralized Training
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- Faster CNNs with Direct Sparse Convolutions and Guided Pruning
- Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
- Loss-aware Weight Quantization of Deep Networks
- Natural Compression for Distributed Deep Learning
- Asynchronous stochastic convex optimization