Decoupled Parallel Backpropagation with Convergence Guarantee

Abstract

Backpropagation algorithm is indispensable for the training of feedforward neural networks. It requires propagating error gradients sequentially from the output layer all the way back to the input layer. The backward locking in backpropagation algorithm constrains us from updating network layers in parallel and fully leveraging the computing resources. Recently, several algorithms have been proposed for breaking the backward locking. However, their performances degrade seriously when networks are deep. In this paper, we propose decoupled parallel backpropagation algorithm for deep learning optimization with convergence guarantee. Firstly, we decouple the backpropagation algorithm using delayed gradients, and show that the backward locking is removed when we split the networks into multiple modules. Then, we utilize decoupled parallel backpropagation in two stochastic methods and prove that our method guarantees convergence to critical points for the non-convex problem. Finally, we perform experiments for training deep convolutional neural networks on benchmark datasets. The experimental results not only confirm our theoretical analysis, but also demonstrate that the proposed method can achieve significant speedup without loss of accuracy.

ICML 2018

Cited by in corpus (14)

• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• FLOP: Federated Learning on Medical Datasets using Partial Networks
• Training Neural Networks Using Features Replay
• XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
• Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• Predict Globally, Correct Locally: Parallel-in-Time Optimal Control of Neural Networks
• Parallel Training of Deep Networks with Local Updates
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• A Practical Layer-Parallel Training Algorithm for Residual Networks
• Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation
• Ouroboros: On Accelerating Training of Transformer-Based Language Models