DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging

Qinggang Zhou, Yawen Zhang, Pengcheng Li +4

The state-of-the-art deep learning algorithms rely on distributed training systems to tackle the increasing sizes of models and training data sets. Minibatch stochastic gradient de…