Fully Decoupled Neural Network Learning Using Delayed Gradients

Huiping Zhuang, Yi Wang, Qinglai Liu +2

Training neural networks with back-propagation (BP) requires a sequential passing of activations and gradients, which forces the network modules to work in a synchronous fashion. T…