PipeDream: Fast and Efficient Pipeline Parallel DNN Training
arXiv:1806.03377
Abstract
PipeDream is a Deep Neural Network(DNN) training system for GPUs that parallelizes computation by pipelining execution across multiple machines. Its pipeline parallel computing model avoids the slowdowns faced by data-parallel training when large models and/or limited network bandwidth induce high communication-to-computation ratios. PipeDream reduces communication by up to 95% for large DNNs relative to data-parallel training, and allows perfect overlap of communication and computation. PipeDream keeps all available GPUs productive by systematically partitioning DNN layers among them to balance work and minimize communication, versions model parameters for backward pass correctness, and schedules the forward and backward passes of different inputs in round-robin fashion to optimize "time to target accuracy". Experiments with five different DNNs on two different clusters show that PipeDream is up to 5x faster in time-to-accuracy compared to data-parallel training.
References in corpus (3)
Cited by in corpus (45)
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
- Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
- Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Memory-Efficient Pipeline-Parallel DNN Training
- ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
- Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
- XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
- Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
- TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
- Scalable and Efficient MoE Training for Multitask Multilingual Models
- torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
- DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
- PipeMare: Asynchronous Pipeline Parallel DNN Training
- Training Large Neural Networks with Constant Memory using a New Execution Algorithm
- Efficient Algorithms for Device Placement of DNN Graph Operators
- Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees
- Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
- Hierarchy-Based Algorithms for Minimizing Makespan under Precedence and Communication Constraints
- Pipelined Backpropagation at Scale: Training Large Models without Batches
- Parallel Training of Deep Networks with Local Updates
- Heterogeneity-Aware Asynchronous Decentralized Training
- Faster and Simpler SNN Simulation with Work Queues
- Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
- Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement
- Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks
- Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
- HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow
- Auto-MAP: A DQN Framework for Exploring Distributed Execution Plans for DNN Workloads
- -ML: Mitigating Adversarial Examples via Ensembles of Topologically Manipulated Classifiers
- 8-bit Optimizers via Block-wise Quantization
- A Study of Checkpointing in Large Scale Training of Deep Neural Networks
- FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Framework for Heterogeneous Edge Devices
- Automatic Graph Partitioning for Very Large-scale Deep Learning
- Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGX
- A Practical Layer-Parallel Training Algorithm for Residual Networks
- Asynchronous Optimization Methods for Efficient Training of Deep Neural Networks with Guarantees
- Prune2Edge: A Multi-Phase Pruning Pipelines to Deep Ensemble Learning in IIoT
- Adaptive Braking for Mitigating Gradient Delay
- HetSeq: Distributed GPU Training on Heterogeneous Infrastructure
- Hydra: A System for Large Multi-Model Deep Learning
- Pipelined Training with Stale Weights of Deep Convolutional Neural Networks