CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation
arXiv:2106.10796
Abstract
Communication overhead is the key challenge for distributed training. Gradient compression is a widely used approach to reduce communication traffic. When combining with parallel communication mechanism method like pipeline, gradient compression technique can greatly alleviate the impact of communication overhead. However, there exists two problems of gradient compression technique to be solved. Firstly, gradient compression brings in extra computation cost, which will delay the next training iteration. Secondly, gradient compression usually leads to the decrease of convergence accuracy.
arXiv comment12 pages
References in corpus (17)
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Rethinking the Value of Network Pruning
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Learning both Weights and Connections for Efficient Neural Networks
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Large Batch Training of Convolutional Networks
- Local SGD Converges Fast and Communicates Little
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- Stochastic Gradient Push for Distributed Deep Learning
- Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
- AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
- Communication optimization strategies for distributed deep neural network training: A survey
- Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
- Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
- RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
- TicTac: Accelerating Distributed Deep Learning with Communication Scheduling
- Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees