LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learning

Abstract

This paper targets solving distributed machine learning problems such as federated learning in a communication-efficient fashion. A class of new stochastic gradient descent (SGD) approaches have been developed, which can be viewed as the stochastic generalization to the recently developed lazily aggregated gradient (LAG) method --- justifying the name LASG. LAG adaptively predicts the contribution of each round of communication and chooses only the significant ones to perform. It saves communication while also maintains the rate of convergence. However, LAG only works with deterministic gradients, and applying it to stochastic gradients yields poor performance. The key components of LASG are a set of new rules tailored for stochastic gradients that can be implemented either to save download, upload, or both. The new algorithms adaptively choose between fresh and stale stochastic gradients and have convergence rates comparable to the original SGD. LASG achieves impressive empirical performance --- it typically saves total communication by an order of magnitude.

References in corpus (7)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Sparsified SGD with Memory
• Don't Use Large Mini-Batches, Use Local SGD
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients
• On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization

Cited by in corpus (3)

• VAFL: a Method of Vertical Asynchronous Federated Learning
• Anarchic Federated Learning
• CADA: Communication-Adaptive Distributed Adam