DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

Abstract

It is a challenging task to train large DNN models on sophisticated GPU platforms with diversified interconnect capabilities. Recently, pipelined training has been proposed as an effective approach for improving device utilization. However, there are still several tricky issues to address: improving computing efficiency while ensuring convergence, and reducing memory usage without incurring additional computing costs. We propose DAPPLE, a synchronous training framework which combines data parallelism and pipeline parallelism for large DNN models. It features a novel parallelization strategy planner to solve the partition and placement problems, and explores the optimal hybrid strategy of data and pipeline parallelism. We also propose a new runtime scheduling algorithm to reduce device memory usage, which is orthogonal to re-computation approach and does not come at the expense of training throughput. Experiments show that DAPPLE planner consistently outperforms strategies generated by PipeDream's planner by up to 3.23x under synchronous training scenarios, and DAPPLE runtime outperforms GPipe by 1.6x speedup of training throughput and reduces the memory consumption of 12% at the same time.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• One weird trick for parallelizing convolutional neural networks
• Training Deep Nets with Sublinear Memory Cost
• Horovod: fast and easy distributed deep learning in TensorFlow
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Beyond Data and Model Parallelism for Deep Neural Networks
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Priority-based Parameter Propagation for Distributed DNN Training
• XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
• PipeMare: Asynchronous Pipeline Parallel DNN Training
• Large-Batch Training for LSTM and Beyond

Cited by in corpus (8)

• Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
• Near-Optimal Sparse Allreduce for Distributed Deep Learning
• Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
• Efficient Pipeline Planning for Expedited Distributed DNN Training
• Reducing Energy Bloat in Large Model Training
• Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading
• Auto-MAP: A DQN Framework for Exploring Distributed Execution Plans for DNN Workloads