Efficient Pipeline Planning for Expedited Distributed DNN Training

Ziyue Luo, Xiaodong Yi, Guoping Long +4

To train modern large DNN models, pipeline parallelism has recently emerged, which distributes the model across GPUs and enables different devices to process different microbatches…