A Double Residual Compression Algorithm for Efficient Distributed Learning

Abstract

Large-scale machine learning models are often trained by parallel stochastic gradient descent algorithms. However, the communication cost of gradient aggregation and model synchronization between the master and worker nodes becomes the major obstacle for efficient learning as the number of workers and the dimension of the model increase. In this paper, we propose DORE, a DOuble REsidual compression stochastic gradient descent algorithm, to reduce over $95\%$ of the overall communication such that the obstacle can be immensely mitigated. Our theoretical analyses demonstrate that the proposed strategy has superior convergence properties for both strongly convex and nonconvex objective functions. The experimental results validate that DORE achieves the best communication efficiency while maintaining similar model accuracy and convergence speed in comparison with start-of-the-art baselines.

References in corpus (12)

• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Sparsified SGD with Memory
• Communication Compression for Decentralized Training
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• Distributed Learning with Compressed Gradient Differences
• SGD and Hogwild! Convergence Without the Bounded Gradients Assumption
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• SGD: General Analysis and Improved Rates

Cited by in corpus (9)

• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Compressed Gradient Tracking for Decentralized Optimization Over General Directed Networks
• Recent theoretical advances in decentralized distributed convex optimization
• BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning
• FedNL: Making Newton-Type Methods Applicable to Federated Learning
• Linear Convergent Decentralized Optimization with Compression
• Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees
• Communication-Efficient Distributed SGD with Error-Feedback, Revisited
• MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization