PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction

Lei Guan, Dongsheng Li, Yongle Chen +3

Asynchronous pipeline model parallelism with a "1F1B" (one forward, one backward) schedule generates little bubble overhead and always provides quite a high throughput. However, th…