DistSim: A performance model of large-scale hybrid distributed DNN training

Guandong Lu, Runzhe Chen, Yakai Wang +8

With the ever-increasing computational demand of DNN training workloads, distributed training has been widely adopted. A combination of data, model and pipeline parallelism strateg…