Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
arXiv:2103.03239
Abstract
Training deep neural networks on large datasets can often be accelerated by using multiple compute nodes. This approach, known as distributed training, can utilize hundreds of computers via specialized message-passing protocols such as Ring All-Reduce. However, running these protocols at scale requires reliable high-speed networking that is only available in dedicated clusters. In contrast, many real-world applications, such as federated learning and cloud-based distributed training, operate on unreliable devices with unstable network bandwidth. As a result, these applications are restricted to using parameter servers or gossip-based averaging protocols. In this work, we lift that restriction by proposing Moshpit All-Reduce - an iterative averaging protocol that exponentially converges to the global average. We demonstrate the efficiency of our protocol for distributed optimization with strong theoretical guarantees. The experiments show 1.3x speedup for ResNet-50 training on ImageNet compared to competitive gossip-based strategies and 1.5x speedup when training ALBERT-large from scratch using preemptible compute nodes.
arXiv commentAccepted to Conference on Neural Information Processing Systems (NeurIPS) 2021. Code: https://github.com/yandex-research/moshpit-sgd
References in corpus (42)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Language Models are Few-Shot Learners
- Federated Learning: Strategies for Improving Communication Efficiency
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Scaling Laws for Neural Language Models
- Federated Learning for Mobile Keyboard Prediction
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- A Survey on Distributed Machine Learning
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Applied Federated Learning: Improving Google Keyboard Query Suggestions
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates
- Sparsified SGD with Memory
- Don't Use Large Mini-Batches, Use Local SGD
- Optimal algorithms for smooth and strongly convex distributed optimization in networks
- Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
- A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- Distributed Mean Estimation with Limited Communication
- An Exact Quantized Decentralized Gradient Descent Algorithm
- Stochastic Gradient Push for Distributed Deep Learning
- Communication Complexity of Distributed Convex Learning and Optimization
- Distributed Delayed Stochastic Optimization
- Distributed Learning with Compressed Gradient Differences
- Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
- Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
- Natural Compression for Distributed Deep Learning
- Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
- On Biased Compression for Distributed Learning
- Minibatch vs Local SGD for Heterogeneous Distributed Learning
- Linearly Converging Error Compensated SGD
- A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization
- Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks
- A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates
- Projected Gradient Method for Decentralized Optimization over Time-Varying Networks
- MARINA: Faster Non-Convex Distributed Learning with Compression
- NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
- Performance Analysis and Comparison of Distributed Machine Learning Systems
- Error Compensated Distributed SGD Can Be Accelerated
- Multi-node Bert-pretraining: Cost-efficient Approach