Double Quantization for Communication-Efficient Distributed Optimization
arXiv:1805.10111
Abstract
Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propose \emph{double quantization}, a general scheme for quantizing both model parameters and gradients. Three communication-efficient algorithms are proposed under this general scheme. Specifically, (i) we propose a low-precision algorithm AsyLPG with asynchronous parallelism, (ii) we explore integrating gradient sparsification with double quantization and develop Sparse-AsyLPG, (iii) we show that double quantization can also be accelerated by momentum technique and design accelerated AsyLPG. We establish rigorous performance guarantees for the algorithms, and conduct experiments on a multi-server test-bed to demonstrate that our algorithms can effectively save transmitted bits without performance degradation.
Cited by in corpus (11)
- Communication optimization strategies for distributed deep neural network training: A survey
- 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
- DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning
- COKE: Communication-Censored Decentralized Kernel Learning
- APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm
- 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
- Localization-aware Channel Pruning for Object Detection
- ErrorCompensatedX: error compensation for variance reduced algorithms
- An Expectation-Maximization Perspective on Federated Learning
- Distributed Sparse SGD with Majority Voting
- A General Framework for Analyzing Stochastic Dynamics in Learning Algorithms