A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning

Abstract

With huge amounts of training data, deep learning has made great breakthroughs in many artificial intelligence (AI) applications. However, such large-scale data sets present computational challenges, requiring training to be distributed on a cluster equipped with accelerators like GPUs. With the fast increase of GPU computing power, the data communications among GPUs have become a potential bottleneck on the overall training performance. In this paper, we first propose a general directed acyclic graph (DAG) model to describe the distributed synchronous stochastic gradient descent (S-SGD) algorithm, which has been widely used in distributed deep learning frameworks. To understand the practical impact of data communications on training performance, we conduct extensive empirical studies on four state-of-the-art distributed deep learning frameworks (i.e., Caffe-MPI, CNTK, MXNet and TensorFlow) over multi-GPU and multi-node environments with different data communication techniques, including PCIe, NVLink, 10GbE, and InfiniBand. Through both analytical and experimental studies, we identify the potential bottlenecks and overheads that could be further optimized. At last, we make the data set of our experimental traces publicly available, which could be used to support simulation-based studies.

arXiv comment8 pages. Accepted by ICPADS'2018

References in corpus (12)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• cuDNN: Efficient Primitives for Deep Learning
• Revisiting Distributed Synchronous SGD
• Large Batch Training of Convolutional Networks
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Comparative Study of Deep Learning Software Frameworks
• Benchmarking State-of-the-Art Deep Learning Software Tools
• ImageNet Training in Minutes
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• Stochastic Nonconvex Optimization with Large Minibatches
• Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?

Cited by in corpus (6)

• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection
• Bandwidth Reduction using Importance Weighted Pruning on Ring AllReduce
• Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
• Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications
• MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning