Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM
arXiv:2009.06459
Abstract
In this paper, we propose a communication-efficiently decentralized machine learning framework that solves a consensus optimization problem defined over a network of inter-connected workers. The proposed algorithm, Censored and Quantized Generalized GADMM (CQ-GGADMM), leverages the worker grouping and decentralized learning ideas of Group Alternating Direction Method of Multipliers (GADMM), and pushes the frontier in communication efficiency by extending its applicability to generalized network topologies, while incorporating link censoring for negligible updates after quantization. We theoretically prove that CQ-GGADMM achieves the linear convergence rate when the local objective functions are strongly convex under some mild assumptions. Numerical simulations corroborate that CQ-GGADMM exhibits higher communication efficiency in terms of the number of communication rounds and transmit energy consumption without compromising the accuracy and convergence speed, compared to the censored decentralized ADMM, and the worker grouping method of GADMM.
arXiv comment14 pages, 5 figures
References in corpus (10)
- Federated Optimization: Distributed Machine Learning for On-Device Intelligence
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
- Sparsified SGD with Memory
- On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
- Extreme URLLC: Vision, Challenges, and Key Enablers
- Understanding the Role of Momentum in Stochastic Gradient Methods
- Federated Learning With Quantized Global Model Updates
- SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimization
- Adaptive Serverless Learning