torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
arXiv:2004.09910
Abstract
We design and implement a ready-to-use library in PyTorch for performing micro-batch pipeline parallelism with checkpointing proposed by GPipe (Huang et al., 2019). In particular, we develop a set of design components to enable pipeline-parallel gradient computation in PyTorch's define-by-run and eager execution environment. We show that each component is necessary to fully benefit from pipeline parallelism in such environment, and demonstrate the efficiency of the library by applying it to various network architectures including AmoebaNet-D and U-Net. Our library is available at https://github.com/kakaobrain/torchgpipe .
10 pages, 7 figures
References in corpus (11)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- One weird trick for parallelizing convolutional neural networks
- Training Deep Nets with Sublinear Memory Cost
- Device Placement Optimization with Reinforcement Learning
- Learning the Number of Neurons in Deep Networks
- Measuring the Effects of Data Parallelism on Neural Network Training
- PipeDream: Fast and Efficient Pipeline Parallel DNN Training
- XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
- GDP: Generalized Device Placement for Dataflow Graphs
Cited by in corpus (8)
- Beyond English-Centric Multilingual Machine Translation
- Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
- PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers
- Pipeline Parallelism for Inference on Heterogeneous Edge Computing
- An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
- Distributed Training and Optimization Of Neural Networks
- Automatic Graph Partitioning for Very Large-scale Deep Learning
- Analyzing the Performance of Graph Neural Networks with Pipe Parallelism