Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency

Abstract

Large-scale language models have become increasingly challenging and expensive to train. Among various methods addressing this issue, Pipeline Parallelism has been widely employed to accommodate massive model weights within limited GPU memory. This paper introduces Hanayo, a wave-like pipeline parallelism strategy that boasts a concise structure and practical applicability, alongside a high-performance pipeline execution runtime to tackle the challenges of pipeline strategy implementation. Hanayo mitigates the issues of pipeline bubbles and excessive memory consumption prevalent in existing schemes, without resorting to model duplicates as in Chimera. Our evaluation, conducted on four distinct computing clusters and involving both GPT-like and BERT-like architectures with up to 32 GPUs, demonstrates up to a 30.4 \% increase in throughput compared to the state-of-the-art approach.

12 pages, 12 figures. This paper has been accepted by SC23(The International Conference for High Performance Computing, Networking, Storage and Analysis). The information can be seen at https://sc23.supercomputing.org/presentation/?id=pap352&sess=sess168

References in corpus (5)

• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
• torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• PipeMare: Asynchronous Pipeline Parallel DNN Training