Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning

Abstract

The high cost of communicating gradients is a major bottleneck for federated learning, as the bandwidth of the participating user devices is limited. Existing gradient compression algorithms are mainly designed for data centers with high-speed network and achieve $O(\sqrt{d}\log d)$ per-iteration communication cost at best, where $d$ is the size of the model. We propose hyper-sphere quantization (HSQ), a general framework that can be configured to achieve a continuum of trade-offs between communication efficiency and gradient accuracy. In particular, at the high compression ratio end, HSQ provides a low per-iteration communication cost of $O(\log d)$, which is favorable for federated learning. We prove the convergence of HSQ theoretically and show by experiments that HSQ significantly reduces the communication cost of model training without hurting convergence accuracy.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Sparsified SGD with Memory
• Communication Compression for Decentralized Training
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Distributed Mean Estimation with Limited Communication
• Stochastic Gradient Push for Distributed Deep Learning
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• Distributed Learning with Sublinear Communication

Cited by in corpus (9)

• Personalized Federated Learning: A Meta-Learning Approach
• Personalized Federated Learning with Moreau Envelopes
• Communication Efficiency in Federated Learning: Achievements and Challenges
• Personalized Federated Learning using Hypernetworks
• Bayesian Federated Learning over Wireless Networks
• Personalized Federated Learning with Gaussian Processes
• Distributed Additive Encryption and Quantization for Privacy Preserving Federated Deep Learning
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection
• Towards Heterogeneous Clients with Elastic Federated Learning