Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training

Abstract

Deploying deep learning (DL) models across multiple compute devices to train large and complex models continues to grow in importance because of the demand for faster and more frequent training. Data parallelism (DP) is the most widely used parallelization strategy, but as the number of devices in data parallel training grows, so does the communication overhead between devices. Additionally, a larger aggregate batch size per step leads to statistical efficiency loss, i.e., a larger number of epochs are required to converge to a desired accuracy. These factors affect overall training time and beyond a certain number of devices, the speedup from leveraging DP begins to scale poorly. In addition to DP, each training step can be accelerated by exploiting model parallelism (MP). This work explores hybrid parallelization, where each data parallel worker is comprised of more than one device, across which the model dataflow graph (DFG) is split using MP. We show that at scale, hybrid training will be more effective at minimizing end-to-end training time than exploiting DP alone. We project that for Inception-V3, GNMT, and BigLSTM, the hybrid strategy provides an end-to-end training speedup of at least 26.5%, 8%, and 22% respectively compared to what DP alone can achieve at scale.

References in corpus (18)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• cuDNN: Efficient Primitives for Deep Learning
• One weird trick for parallelizing convolutional neural networks
• Exploring the Limits of Language Modeling
• Revisiting Distributed Synchronous SGD
• Don't Decay the Learning Rate, Increase the Batch Size
• Horovod: fast and easy distributed deep learning in TensorFlow
• Three Factors Influencing Minima in SGD
• Device Placement Optimization with Reinforcement Learning
• Measuring the Effects of Data Parallelism on Neural Network Training
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training
• Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers