Decentralized Deep Learning with Arbitrary Communication Compression

Abstract

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches suffer from limited bandwidth of the network, we propose the use of communication compression in the decentralized training context. We show that Choco-SGD $-$ recently introduced and analyzed for strongly-convex objectives only $-$ converges under arbitrary high compression ratio on general non-convex functions at the rate $O(1/\sqrt{nT})$ where $T$ denotes the number of iterations and $n$ the number of workers. The algorithm achieves linear speedup in the number of workers and supports higher compression than previous state-of-the art methods. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models (i) over distributed user devices, connected by a social network and (ii) in a datacenter (outperforming all-reduce time-wise).

References in corpus (28)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Optimal Distributed Online Prediction using Mini-Batches
• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models
• Sparsified SGD with Memory
• Don't Use Large Mini-Batches, Use Local SGD
• Distributed Coordinate Descent Method for Learning with Big Data
• Optimal algorithms for smooth and strongly convex distributed optimization in networks
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Communication Compression for Decentralized Training
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• An Exact Quantized Decentralized Gradient Descent Algorithm
• Stochastic Gradient Push for Distributed Deep Learning
• Distributed Learning with Compressed Gradient Differences
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• Parallel SGD: When does averaging help?
• Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
• $\text{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression
• Approximate Steepest Coordinate Descent
• Robust and Communication-Efficient Collaborative Learning
• Nested Distributed Gradient Methods with Adaptive Quantized Communication

Cited by in corpus (17)

• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
• Communication Efficiency in Federated Learning: Achievements and Challenges
• A Decentralized Parallel Algorithm for Training Generative Adversarial Nets
• Linearly Converging Error Compensated SGD
• Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk Minimization
• GFL: A Decentralized Federated Learning Framework Based On Blockchain
• SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimization
• Gradient Descent with Compressed Iterates
• Trends and Advancements in Deep Neural Network Communication
• Adaptive Serverless Learning
• On the Convergence of Nested Decentralized Gradient Methods with Multiple Consensus and Gradient Steps
• Decentralized Optimization On Time-Varying Directed Graphs Under Communication Constraints
• Cross-Gradient Aggregation for Decentralized Learning from Non-IID data
• Boosting Asynchronous Decentralized Learning with Model Fragmentation
• Machine Learning Systems for Intelligent Services in the IoT: A Survey
• CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
• Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization