ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation

Abstract

Training large Deep Neural Network (DNN) models requires thousands of GPUs over the course of several days or weeks. At this scale, failures are frequent and can have a big impact on training throughput. Utilizing spare GPU servers to mitigate performance loss becomes increasingly costly as model sizes grow. ReCycle is a system designed for efficient DNN training in the presence of failures, without relying on spare servers. It exploits the inherent functional redundancy in distributed training systems -- where servers across data-parallel groups store the same model parameters -- and pipeline schedule bubbles within each data-parallel group. When servers fails, ReCycle dynamically re-routes micro-batches to data-parallel peers, allowing for uninterrupted training despite multiple failures. However, this re-routing can create imbalances across pipeline stages, leading to reduced training throughput. To address this, ReCycle introduces two key optimizations that ensure re-routed micro-batches are processed within the original pipeline schedule's bubbles. First, it decouples the backward pass into two phases: one for computing gradients for the input and another for calculating gradients for the parameters. Second, it avoids synchronization across pipeline stages by staggering the optimizer step. Together, these optimizations enable adaptive pipeline schedules that minimize or even eliminate training throughput degradation during failures. We describe a prototype for ReCycle and show that it achieves high training throughput under multiple failures, outperforming recent proposals for fault-tolerant training such as Oobleck and Bamboo by up to $1.46\times$ and $1.64\times$, respectively.

SOSP'24 | Camera-Ready

References in corpus (24)

• Decoupled Weight Decay Regularization
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On the Opportunities and Risks of Foundation Models
• Scaling Laws for Neural Language Models
• One weird trick for parallelizing convolutional neural networks
• Training Compute-Optimal Large Language Models
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
• Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
• TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism
• ImageNet Training in Minutes
• Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates
• Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
• Machine Learning Model Sizes and the Parameter Gap
• TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
• Detecting silent data corruptions in the wild
• Data Dropout: Optimizing Training Data for Convolutional Neural Networks
• Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• Unicron: Economizing Self-Healing LLM Training at Scale
• Revisiting Distributed Synchronous SGD

Cited by in corpus (1)

• DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline