Fully Decoupled Neural Network Learning Using Delayed Gradients

Abstract

Training neural networks with back-propagation (BP) requires a sequential passing of activations and gradients, which forces the network modules to work in a synchronous fashion. This has been recognized as the lockings (i.e., the forward, backward and update lockings) inherited from the BP. In this paper, we propose a fully decoupled training scheme using delayed gradients (FDG) to break all these lockings. The FDG splits a neural network into multiple modules and trains them independently and asynchronously using different workers (e.g., GPUs). We also introduce a gradient shrinking process to reduce the stale gradient effect caused by the delayed gradients. In addition, we prove that the proposed FDG algorithm guarantees a statistical convergence during training. Experiments are conducted by training deep convolutional neural networks to perform classification tasks on benchmark datasets, showing comparable or better results against the state-of-the-art methods as well as the BP in terms of both generalization and acceleration abilities. In particular, we show that the FDG is also able to train very wide networks (e.g., WRN-28-10) and extremely deep networks (e.g., ResNet-1202). Code is available at https://github.com/ZHUANGHP/FDG.

References in corpus (12)

• Wide Residual Networks
• Revisiting Distributed Synchronous SGD
• Horovod: fast and easy distributed deep learning in TensorFlow
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Shake-Shake regularization
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Beyond Data and Model Parallelism for Deep Neural Networks
• Training Neural Networks with Local Error Signals
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Decoupled Neural Interfaces using Synthetic Gradients
• Decoupled Parallel Backpropagation with Convergence Guarantee
• Training Neural Networks Using Features Replay

Cited by in corpus (4)

• Pipelined Backpropagation at Scale: Training Large Models without Batches
• Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework
• Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization
• Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework