Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback

Abstract

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote achieves a 32x reduction on communication cost. However, its convergence is based on unrealistic assumptions and can diverge in practice. In this paper, we propose a general distributed compressed SGD with Nesterov's momentum. We consider two-way compression, which compresses the gradients both to and from workers. Convergence analysis on nonconvex problems for general gradient compressors is provided. By partitioning the gradient into blocks, a blockwise compressor is introduced such that each gradient block is compressed and transmitted in 1-bit format with a scaling factor, leading to a nearly 32x reduction on communication. Experimental results show that the proposed method converges as fast as full-precision distributed momentum SGD and achieves the same testing accuracy. In particular, on distributed ResNet training with 7 workers on the ImageNet, the proposed algorithm achieves the same testing accuracy as momentum SGD using full-precision gradients, but with $46\%$ less wall clock time.

arXiv commentNeurIPS 2019

References in corpus (9)

• Recurrent Neural Network Regularization
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Revisiting Distributed Synchronous SGD
• Sparsified SGD with Memory
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression

Cited by in corpus (15)

• High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• FetchSGD: Communication-Efficient Federated Learning with Sketching
• Optimal Gradient Compression for Distributed and Federated Learning
• Auto-weighted Robust Federated Learning with Corrupted Data Sources
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
• On the Utility of Gradient Compression in Distributed Training Systems
• Communication-Efficient Distributed SGD with Error-Feedback, Revisited
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
• Compressing gradients by exploiting temporal correlation in momentum-SGD
• Communication-Efficient Zeroth-Order Distributed Online Optimization: Algorithm, Theory, and Applications
• Quantized Adam with Error Feedback
• CSER: Communication-efficient SGD with Error Reset
• Communication Efficient Generalized Tensor Factorization for Decentralized Healthcare Networks