Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
arXiv:2103.03239
Abstract
Training deep neural networks on large datasets can often be accelerated by using multiple compute nodes. This approach, known as distributed training, can utilize hundreds of computers via specialized message-passing protocols such as Ring All-Reduce. However, running these protocols at scale requires reliable high-speed networking that is only available in dedicated clusters. In contrast, many real-world applications, such as federated learning and cloud-based distributed training, operate on unreliable devices with unstable network bandwidth. As a result, these applications are restricted to using parameter servers or gossip-based averaging protocols. In this work, we lift that restriction by proposing Moshpit All-Reduce - an iterative averaging protocol that exponentially converges to the global average. We demonstrate the efficiency of our protocol for distributed optimization with strong theoretical guarantees. The experiments show 1.3x speedup for ResNet-50 training on ImageNet compared to competitive gossip-based strategies and 1.5x speedup when training ALBERT-large from scratch using preemptible compute nodes.
Accepted to Conference on Neural Information Processing Systems (NeurIPS) 2021. Code: https://github.com/yandex-research/moshpit-sgd
References in corpus (22)
- Language Models are Few-Shot Learners
- Scaling Laws for Neural Language Models
- A Survey on Distributed Machine Learning
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
- A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- Distributed Delayed Stochastic Optimization
- Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
- Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
- On Biased Compression for Distributed Learning
- Minibatch vs Local SGD for Heterogeneous Distributed Learning
- Linearly Converging Error Compensated SGD
- Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks
- A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization
- Projected Gradient Method for Decentralized Optimization over Time-Varying Networks
- MARINA: Faster Non-Convex Distributed Learning with Compression
- NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
- Performance Analysis and Comparison of Distributed Machine Learning Systems
- Error Compensated Distributed SGD Can Be Accelerated
- Multi-node Bert-pretraining: Cost-efficient Approach