Shuffle-Exchange Brings Faster: Reduce the Idle Time During Communication for Decentralized Neural Network Training
arXiv:2007.00433
Abstract
As a crucial scheme to accelerate the deep neural network (DNN) training, distributed stochastic gradient descent (DSGD) is widely adopted in many real-world applications. In most distributed deep learning (DL) frameworks, DSGD is implemented with Ring-AllReduce architecture (Ring-SGD) and uses a computation-communication overlap strategy to address the overhead of the massive communications required by DSGD. However, we observe that although gradients are needed to be communicated per worker in Ring-SGD, the handshakes required by Ring-SGD limits its usage when training with many workers or in high latency network. In this paper, we propose Shuffle-Exchange SGD (SESGD) to solve the dilemma of Ring-SGD. In the cluster of 16 workers with 0.1ms Ethernet latency, SESGD can accelerate the DNN training to without losing model accuracy. Moreover, the process can be accelerated up to in high latency networks (5ms).
arXiv commentNeurIPS 2020 Under Review
References in corpus (7)
- Federated Learning: Strategies for Improving Communication Efficiency
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Horovod: fast and easy distributed deep learning in TensorFlow
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Don't Use Large Mini-Batches, Use Local SGD
- Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
- Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent