DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging
arXiv:2006.00441
Abstract
The state-of-the-art deep learning algorithms rely on distributed training systems to tackle the increasing sizes of models and training data sets. Minibatch stochastic gradient descent (SGD) algorithm requires workers to halt forward/back propagations, to wait for gradients aggregated from all workers, and to receive weight updates before the next batch of tasks. This synchronous execution model exposes the overheads of gradient/weight communication among a large number of workers in a distributed training system. We propose a new SGD algorithm, DaSGD (Local SGD with Delayed Averaging), which parallelizes SGD and forward/back propagations to hide 100% of the communication overhead. By adjusting the gradient update scheme, this algorithm uses hardware resources more efficiently and reduces the reliance on the low-latency and high-throughput inter-connects. The theoretical analysis and the experimental results show its convergence rate O(1/sqrt(K)), the same as SGD. The performance evaluation demonstrates it enables a linear performance scale-up with the cluster size.
References in corpus (15)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Optimal Distributed Online Prediction using Mini-Batches
- Large Batch Training of Convolutional Networks
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
- A Reliable Effective Terascale Linear Learning System
- Don't Use Large Mini-Batches, Use Local SGD
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- Image Classification at Supercomputer Scale