Toward Communication Efficient Adaptive Gradient Method

Abstract

In recent years, distributed optimization is proven to be an effective approach to accelerate training of large scale machine learning models such as deep neural networks. With the increasing computation power of GPUs, the bottleneck of training speed in distributed training is gradually shifting from computation to communication. Meanwhile, in the hope of training machine learning models on mobile devices, a new distributed training paradigm called ``federated learning'' has become popular. The communication time in federated learning is especially important due to the low bandwidth of mobile devices. While various approaches to improve the communication efficiency have been proposed for federated learning, most of them are designed with SGD as the prototype training algorithm. While adaptive gradient methods have been proven effective for training neural nets, the study of adaptive gradient methods in federated learning is scarce. In this paper, we propose an adaptive gradient method that can guarantee both the convergence and the communication efficiency for federated learning.

References in corpus (16)

• ADADELTA: An Adaptive Learning Rate Method
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• On the Convergence of Adam and Beyond
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Improving Generalization Performance by Switching from Adam to SGD
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• Escaping Saddle Points with Adaptive Gradient Methods
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
• FedSKETCH: Communication-Efficient and Private Federated Learning via Sketching
• Efficient Full-Matrix Adaptive Regularization
• On the Convergence of Decentralized Adaptive Gradient Methods