Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
arXiv:1806.08054
Abstract
Large-scale distributed optimization is of great importance in various applications. For data-parallel based distributed learning, the inter-node gradient communication often becomes the performance bottleneck. In this paper, we propose the error compensated quantized stochastic gradient descent algorithm to improve the training efficiency. Local gradients are quantized to reduce the communication overhead, and accumulated quantization error is utilized to speed up the convergence. Furthermore, we present theoretical analysis on the convergence behaviour, and demonstrate its advantage over competitors. Extensive experiments indicate that our algorithm can compress gradients by a factor of up to two magnitudes without performance degradation.
arXiv commentAccepted by ICML 2018
References in corpus (2)
Cited by in corpus (14)
- Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
- Fast Federated Learning by Balancing Communication Trade-Offs
- On-Device Machine Learning: An Algorithms and Learning Theory Perspective
- An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
- Federated Few-Shot Learning for Mobile NLP
- FedSKETCH: Communication-Efficient and Private Federated Learning via Sketching
- Faster Non-Convex Federated Learning via Global and Local Momentum
- Towards Practical Few-shot Federated NLP
- Optimal Gradient Quantization Condition for Communication-Efficient Distributed Training
- Accelerated Sparsified SGD with Error Feedback
- Trajectory Normalized Gradients for Distributed Optimization
- A Survey on Large-scale Machine Learning
- ErrorCompensatedX: error compensation for variance reduced algorithms
- MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training