OptPipe: Memory- and Scheduling-Optimized Pipeline Parallelism for LLM Training

Hongpei Li, Han Zhang, Huikang Liu +2

Pipeline parallelism (PP) has become a standard technique for scaling large language model (LLM) training across multiple devices. However, despite recent progress in reducing memo…