DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

Shiqing Fan, Yi Rong, Chen Meng +10

It is a challenging task to train large DNN models on sophisticated GPU platforms with diversified interconnect capabilities. Recently, pipelined training has been proposed as an e…