Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
arXiv:2010.10458
Abstract
Distributed training techniques have been widely deployed in large-scale deep neural networks (DNNs) training on dense-GPU clusters. However, on public cloud clusters, due to the moderate inter-connection bandwidth between instances, traditional state-of-the-art distributed training systems cannot scale well in training large-scale models. In this paper, we propose a new computing and communication efficient top-k sparsification communication library for distributed training. To further improve the system scalability, we optimize I/O by proposing a simple yet efficient multi-level data caching mechanism and optimize the update operation by introducing a novel parallel tensor operator. Experimental results on a 16-node Tencent Cloud cluster (each node with 8 Nvidia Tesla V100 GPUs) show that our system achieves 25%-40% faster than existing state-of-the-art systems on CNNs and Transformer. We finally break the record on DAWNBench on training ResNet-50 to 93% top-5 accuracy on ImageNet.
arXiv comment13 pages
References in corpus (11)
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Sparsified SGD with Memory
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
- Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
- Understanding Top-k Sparsification in Distributed Deep Learning
- RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
- Blink: Fast and Generic Collectives for Distributed ML
- AdaScale SGD: A User-Friendly Algorithm for Distributed Training
Cited by in corpus (4)
- An Overview of Federated Learning at the Edge and Distributed Ledger Technologies for Robotic and Autonomous Systems
- Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression
- Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
- Flexible Communication for Optimal Distributed Learning over Unpredictable Networks