CSER: Communication-efficient SGD with Error Reset
arXiv:2007.13221
Abstract
The scalability of Distributed Stochastic Gradient Descent (SGD) is today limited by communication bottlenecks. We propose a novel SGD variant: Communication-efficient SGD with Error Reset, or CSER. The key idea in CSER is first a new technique called "error reset" that adapts arbitrary compressors for SGD, producing bifurcated local models with periodic reset of resulting local residual errors. Second we introduce partial synchronization for both the gradients and the models, leveraging advantages from them. We prove the convergence of CSER for smooth non-convex problems. Empirical results show that when combined with highly aggressive compressors, the CSER algorithms accelerate the distributed training by nearly 10x for CIFAR-100, and by 4.5x for ImageNet.
References in corpus (9)
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Horovod: fast and easy distributed deep learning in TensorFlow
- Sparsified SGD with Memory
- Don't Use Large Mini-Batches, Use Local SGD
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
- Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback