Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent

Shuheng Shen, Linli Xu, Jingchang Liu +2

With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization ch…