OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training

Yemao Xu, Dezun Dong, Weixia Xu +1

The training of modern deep learning neural network calls for large amounts of computation, which is often provided by GPUs or other specific accelerators. To scale out to achieve…