Beyond Data and Model Parallelism for Deep Neural Networks

Abstract

The computational requirements for training deep neural networks (DNNs) have grown to the point that it is now standard practice to parallelize training. Existing deep learning systems commonly use data or model parallelism, but unfortunately, these strategies often result in suboptimal parallelization performance. In this paper, we define a more comprehensive search space of parallelization strategies for DNNs called SOAP, which includes strategies to parallelize a DNN in the Sample, Operation, Attribute, and Parameter dimensions. We also propose FlexFlow, a deep learning framework that uses guided randomized search of the SOAP space to find a fast parallelization strategy for a specific parallel machine. To accelerate this search, FlexFlow introduces a novel execution simulator that can accurately predict a parallelization strategy's performance and is three orders of magnitude faster than prior approaches that have to execute each strategy. We evaluate FlexFlow with six real-world DNN benchmarks on two GPU clusters and show that FlexFlow can increase training throughput by up to 3.8x over state-of-the-art approaches, even when including its search time, and also improves scalability.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Recurrent Neural Network Regularization
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Densely Connected Convolutional Networks
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Going Deeper with Convolutions
• One weird trick for parallelizing convolutional neural networks
• Device Placement Optimization with Reinforcement Learning
• Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks

Cited by in corpus (36)

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• Mesh-TensorFlow: Deep Learning for Supercomputers
• TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism
• Reinforced Genetic Algorithm Learning for Optimizing Computation Graphs
• Exascale Deep Learning for Scientific Inverse Problems
• High Resolution Medical Image Analysis with Spatial Partitioning
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks
• On the Utility of Gradient Compression in Distributed Training Systems
• Pre-Trained Models: Past, Present and Future
• FuncPipe: A Pipelined Serverless Framework for Fast and Cost-efficient Training of Deep Learning Models
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Taming Momentum in a Distributed Asynchronous Environment
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
• MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions
• Model-Parallel Model Selection for Deep Learning Systems
• Heterogeneity-Aware Asynchronous Decentralized Training
• Neural Architecture Search using Property Guided Synthesis
• Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement
• CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs
• Auto-MAP: A DQN Framework for Exploring Distributed Execution Plans for DNN Workloads
• VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware
• Performance Analysis and Characterization of Training Deep Learning Models on Mobile Devices
• TAGC: Optimizing Gradient Communication in Distributed Transformer Training
• Federated Learning is Better with Non-Homomorphic Encryption
• RLgraph: Modular Computation Graphs for Deep Reinforcement Learning
• Simulating Performance of ML Systems with Offline Profiling
• Transferable Graph Optimizers for ML Compilers
• Hydra: A System for Large Multi-Model Deep Learning
• Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization
• Brief Announcement: On the Limits of Parallelizing Convolutional Neural Networks on GPUs