PipeMare: Asynchronous Pipeline Parallel DNN Training

Abstract

Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing PP techniques sacrifice hardware efficiency by decreasing pipeline utilization or incurring extra memory costs. In this paper, we investigate to what extent these sacrifices are necessary. We devise PipeMare, a simple yet robust training method that tolerates asynchronous updates during PP execution without sacrificing utilization or memory, which allows efficient use of fine-grained pipeline parallelism. Concretely, when tested on ResNet and Transformer networks, asynchrony enables PipeMare to use up to $2.7\times$ less memory or get $4.3\times$ higher pipeline utilization, with similar model quality, when compared to state-of-the-art synchronous PP training techniques.

References in corpus (5)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Training Deep Nets with Sublinear Memory Cost
• Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training

Cited by in corpus (10)

• Memory-Efficient Pipeline-Parallel DNN Training
• Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
• Parallel Training of Deep Networks with Local Updates
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• Auto-MAP: A DQN Framework for Exploring Distributed Execution Plans for DNN Workloads
• Tensor Relational Algebra for Machine Learning System Design
• Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
• Adaptive Braking for Mitigating Gradient Delay