CADA: Communication-Adaptive Distributed Adam

Abstract

Stochastic gradient descent (SGD) has taken the stage as the primary workhorse for large-scale machine learning. It is often used with its adaptive variants such as AdaGrad, Adam, and AMSGrad. This paper proposes an adaptive stochastic gradient descent method for distributed machine learning, which can be viewed as the communication-adaptive counterpart of the celebrated Adam method - justifying its name CADA. The key components of CADA are a set of new rules tailored for adaptive stochastic gradients that can be implemented to save communication upload. The new algorithms adaptively reuse the stale Adam gradients, thus saving communication, and still have convergence rates comparable to original Adam. In numerical experiments, CADA achieves impressive empirical performance in terms of total communication round reduction.

OPT2020: NeurIPS Workshop on Optimization for Machine Learning

References in corpus (7)

• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learning
• APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm