Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent

Abstract

With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization challenges. Nevertheless, although distributed stochastic gradient descent (SGD) algorithms can achieve a linear iteration speedup, they are limited significantly in practice by the communication cost, making it difficult to achieve a linear time speedup. In this paper, we propose a computation and communication decoupled stochastic gradient descent (CoCoD-SGD) algorithm to run computation and communication in parallel to reduce the communication cost. We prove that CoCoD-SGD has a linear iteration speedup with respect to the total computation capability of the hardware resources. In addition, it has a lower communication complexity and better time speedup comparing with traditional distributed SGD algorithms. Experiments on deep neural network training demonstrate the significant improvements of CoCoD-SGD: when training ResNet18 and VGG16 with 16 Geforce GTX 1080Ti GPUs, CoCoD-SGD is up to 2-3$\times$ faster than traditional synchronous SGD.

arXiv commentIJCAI2019, 20 pages, 21 figures

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Revisiting Distributed Synchronous SGD
• Horovod: fast and easy distributed deep learning in TensorFlow
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Sparsified SGD with Memory
• Local SGD Converges Fast and Communicates Little
• Distributed Delayed Stochastic Optimization
• Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training

Cited by in corpus (3)

• Variance Reduced Local SGD with Lower Communication Complexity
• Intermittent Pulling with Local Compensation for Communication-Efficient Federated Learning
• Shuffle-Exchange Brings Faster: Reduce the Idle Time During Communication for Decentralized Neural Network Training