OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training
arXiv:2005.06728
Abstract
The training of modern deep learning neural network calls for large amounts of computation, which is often provided by GPUs or other specific accelerators. To scale out to achieve faster training speed, two update algorithms are mainly applied in the distributed training process, i.e. the Synchronous SGD algorithm (SSGD) and Asynchronous SGD algorithm (ASGD). SSGD obtains good convergence point while the training speed is slowed down by the synchronous barrier. ASGD has faster training speed but the convergence point is lower when compared to SSGD. To sufficiently utilize the advantages of SSGD and ASGD, we propose a novel technology named One-step Delay SGD (OD-SGD) to combine their strengths in the training process. Therefore, we can achieve similar convergence point and training speed as SSGD and ASGD separately. To the best of our knowledge, we make the first attempt to combine the features of SSGD and ASGD to improve distributed training performance. Each iteration of OD-SGD contains a global update in the parameter server node and local updates in the worker nodes, the local update is introduced to update and compensate the delayed local weights. We evaluate our proposed algorithm on MNIST, CIFAR-10 and ImageNet datasets. Experimental results show that OD-SGD can obtain similar or even slightly better accuracy than SSGD, while its training speed is much faster, which even exceeds the training speed of ASGD.
References in corpus (17)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Mixed Precision Training
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Revisiting Distributed Synchronous SGD
- Don't Decay the Learning Rate, Increase the Batch Size
- Horovod: fast and easy distributed deep learning in TensorFlow
- Large Batch Training of Convolutional Networks
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
- Asynchronous Stochastic Gradient Descent with Delay Compensation
- SparkNet: Training Deep Networks in Spark
- High-Accuracy Low-Precision Training
- Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
- Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train