Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency

Abstract

Large-scale language models have become increasingly challenging and expensive to train. Among various methods addressing this issue, Pipeline Parallelism has been widely employed to accommodate massive model weights within limited GPU memory. This paper introduces Hanayo, a wave-like pipeline parallelism strategy that boasts a concise structure and practical applicability, alongside a high-performance pipeline execution runtime to tackle the challenges of pipeline strategy implementation. Hanayo mitigates the issues of pipeline bubbles and excessive memory consumption prevalent in existing schemes, without resorting to model duplicates as in Chimera. Our evaluation, conducted on four distinct computing clusters and involving both GPT-like and BERT-like architectures with up to 32 GPUs, demonstrates up to a 30.4 \% increase in throughput compared to the state-of-the-art approach.

arXiv comment12 pages, 12 figures. This paper has been accepted by SC23(The International Conference for High Performance Computing, Networking, Storage and Analysis). The information can be seen at https://sc23.supercomputing.org/presentation/?id=pap352&sess=sess168

References in corpus (16)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Mixed Precision Training
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Training Deep Nets with Sublinear Memory Cost
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Staleness-aware Async-SGD for Distributed Deep Learning
• Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
• ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
• Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
• PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management
• torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
• Tesseract: Parallelize the Tensor Parallelism Efficiently
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• PipeMare: Asynchronous Pipeline Parallel DNN Training
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
• Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training

Cited by in corpus (2)

• ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation
• DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline