Oscars: Adaptive Semi-Synchronous Parallel Model for Distributed Deep Learning with Global View
arXiv:2102.08550
Abstract
Deep learning has become an indispensable part of life, such as face recognition, NLP, etc., but the training of deep model has always been a challenge, and in recent years, the complexity of training data and models has shown explosive growth, so the training method is gradually transformed into distributed training. Classical synchronization strategy can guarantee accuracy but frequent communication can lead to a slow training speed, although asynchronous strategy training speed but can not guarantee the accuracy, and in the face of the training of the heterogeneous cluster, the above work is not efficient to work, on the one hand, can cause serious waste of resources, on the other hand, frequent communication also made slow training speed, so this paper proposes a semi-synchronous training strategy based on local-SDG, effectively improve the utilization efficiency of heterogeneous resources cluster and reduce communication overhead, to accelerate the training and ensure the accuracy of the model.
arXiv commentarXiv admin note: text overlap with arXiv:1910.13598 by other authors
References in corpus (11)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Horovod: fast and easy distributed deep learning in TensorFlow
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Local SGD Converges Fast and Communicates Little
- Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
- Parallel SGD: When does averaging help?
- Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
- On Data Dependence in Distributed Stochastic Optimization