Quantized Adam with Error Feedback
arXiv:2004.14180
Abstract
In this paper, we present a distributed variant of adaptive stochastic gradient method for training deep neural networks in the parameter-server model. To reduce the communication cost among the workers and server, we incorporate two types of quantization schemes, i.e., gradient quantization and weight quantization, into the proposed distributed Adam. Besides, to reduce the bias introduced by quantization operations, we propose an error-feedback technique to compensate for the quantized gradient. Theoretically, in the stochastic nonconvex setting, we show that the distributed adaptive gradient method with gradient quantization and error-feedback converges to the first-order stationary point, and that the distributed adaptive gradient method with weight quantization and error-feedback converges to the point related to the quantized level under both the single-worker and multi-worker modes. At last, we apply the proposed distributed adaptive gradient methods to train deep neural networks. Experimental results demonstrate the efficacy of our methods.
arXiv commentAccepted to ACM Transactions on Intelligent Systems and Technology
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- On the Convergence of Adam and Beyond
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Training and Inference with Integers in Deep Neural Networks
- AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- Adaptive Federated Optimization
- Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
- Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning
- The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
- Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
- Gradient Descent with Compressed Iterates