BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Training

Houming Wu, Ling Chen, Wenjie Yu

With the increasing scale of models, the need for efficient distributed training has become increasingly urgent. Recently, many synchronous pipeline parallelism approaches have bee…