Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks

Abstract

The past few years have witnessed growth in the computational requirements for training deep convolutional neural networks. Current approaches parallelize training onto multiple devices by applying a single parallelization strategy (e.g., data or model parallelism) to all layers in a network. Although easy to reason about, these approaches result in suboptimal runtime performance in large-scale distributed training, since different layers in a network may prefer different parallelization strategies. In this paper, we propose layer-wise parallelism that allows each layer in a network to use an individual parallelization strategy. We jointly optimize how each layer is parallelized by solving a graph search problem. Our evaluation shows that layer-wise parallelism outperforms state-of-the-art approaches by increasing training throughput, reducing communication costs, achieving better scalability to multiple GPUs, while maintaining original network accuracy.

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Going Deeper with Convolutions
• One weird trick for parallelizing convolutional neural networks
• Device Placement Optimization with Reinforcement Learning

Cited by in corpus (16)

• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Beyond Data and Model Parallelism for Deep Neural Networks
• Mesh-TensorFlow: Deep Learning for Supercomputers
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
• On the Utility of Gradient Compression in Distributed Training Systems
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks
• Hot-Starting the Ac Power Flow with Convolutional Neural Networks
• An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
• BaPipe: Exploration of Balanced Pipeline Parallelism for DNN Training
• Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement
• Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
• Energy-Aware DNN Graph Optimization
• A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities
• Scaling Distributed Deep Learning Workloads beyond the Memory Capacity with KARMA
• Creating Robust Deep Neural Networks With Coded Distributed Computing for IoT Systems
• Simulating Performance of ML Systems with Offline Profiling