Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization

Abstract

Due to the explosion in the size of the training datasets, distributed learning has received growing interest in recent years. One of the major bottlenecks is the large communication cost between the central server and the local workers. While error feedback compression has been proven to be successful in reducing communication costs with stochastic gradient descent (SGD), there are much fewer attempts in building communication-efficient adaptive gradient methods with provable guarantees, which are widely used in training large-scale machine learning models. In this paper, we propose a new communication-compressed AMSGrad for distributed nonconvex optimization problem, which is provably efficient. Our proposed distributed learning framework features an effective gradient compression strategy and a worker-side model update design. We prove that the proposed communication-efficient distributed adaptive gradient method converges to the first-order stationary point with the same iteration complexity as uncompressed vanilla AMSGrad in the stochastic nonconvex optimization setting. Experiments on various benchmarks back up our theory.

arXiv commentAccepted by AISTATS 2022 (29 pages, 11 figures, 2 tables)

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Decoupled Weight Decay Regularization
• ADADELTA: An Adaptive Learning Rate Method
• Language Models are Few-Shot Learners
• Wide Residual Networks
• On the Convergence of Adam and Beyond
• Sparsified SGD with Memory
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• On Biased Compression for Distributed Learning
• EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback
• Decentralized Deep Learning with Arbitrary Communication Compression
• Communication-efficient distributed SGD with Sketching
• 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
• Quantized Adam with Error Feedback