Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients

Abstract

The present paper develops a novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication. The key idea is to first quantize the computed gradients, and then skip less informative quantized gradient communications by reusing outdated gradients. Quantizing and skipping result in `lazy' worker-server communications, which justifies the term Lazily Aggregated Quantized gradient that is henceforth abbreviated as LAQ. Our LAQ can provably attain the same linear convergence rate as the gradient descent in the strongly convex case, while effecting major savings in the communication overhead both in transmitted bits as well as in communication rounds. Empirically, experiments with real data corroborate a significant communication reduction compared to existing gradient- and stochastic gradient-based algorithms.

Accepted in NeurIPS 2019

Cited by in corpus (18)

• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning
• 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
• LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learning
• Communication-Efficient Robust Federated Learning Over Heterogeneous Datasets
• Fairness-aware Agnostic Federated Learning
• On the Benefits of Multiple Gossip Steps in Communication-Constrained Decentralized Optimization
• Leveraging Spatial and Temporal Correlations in Sparsified Mean Estimation
• Intermittent Pulling with Local Compensation for Communication-Efficient Federated Learning
• APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm
• Fast Quantized Average Consensus over Static and Dynamic Directed Graphs
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
• ErrorCompensatedX: error compensation for variance reduced algorithms
• Step-Ahead Error Feedback for Distributed Training with Compressed Gradient
• CADA: Communication-Adaptive Distributed Adam
• Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
• L-DQN: An Asynchronous Limited-Memory Distributed Quasi-Newton Method