RedSync : Reducing Synchronization Traffic for Distributed Deep Learning

Abstract

Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes. Since synchronizing a large number of gradients of the local model can be a bottleneck for large-scale distributed training, compressing communication data has gained widespread attention recently. Among several recent proposed compression algorithms, Residual Gradient Compression (RGC) is one of the most successful approaches---it can significantly compress the transmitting message size (0.1\% of the gradient size) of each node and still achieve correct accuracy and the same convergence speed. However, the literature on compressing deep networks focuses almost exclusively on achieving good theoretical compression rate, while the efficiency of RGC in real distributed implementation has been less investigated. In this paper, we develop an RGC-based system that is able to reduce the end-to-end training time on real-world multi-GPU systems. Our proposed design called RedSync, which introduces a set of optimizations to reduce communication bandwidth requirement while introducing limited overhead. We evaluate the performance of RedSync on two different multiple GPU platforms, including 128 GPUs of a supercomputer and an 8-GPU server. Our test cases include image classification tasks on Cifar10 and ImageNet, and language modeling tasks on Penn Treebank and Wiki2 datasets. For DNNs featured with high communication to computation ratio, which have long been considered with poor scalability, RedSync brings significant performance improvements.

arXiv comment10 pages. Journal of Parallel and Distributed Computing, 2019

References in corpus (10)

• An overview of gradient descent optimization algorithms
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Pointer Sentinel Mixture Models
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• SparCML: High-Performance Sparse Communication for Machine Learning

Cited by in corpus (8)

• Understanding Top-k Sparsification in Distributed Deep Learning
• Communication optimization strategies for distributed deep neural network training: A survey
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
• Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
• On the Utility of Gradient Compression in Distributed Training Systems
• Applications of Federated Learning in Smart Cities: Recent Advances, Taxonomy, and Open Challenges
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
• CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation