DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining

Lin Zhang, Shaohuai Shi, Xiaowen Chu +3

Communication scheduling has been shown to be effective in accelerating distributed training, which enables all-reduce communications to be overlapped with backpropagation computat…