Predict Globally, Correct Locally: Parallel-in-Time Optimal Control of Neural Networks

Abstract

The links between optimal control of dynamical systems and neural networks have proved beneficial both from a theoretical and from a practical point of view. Several researchers have exploited these links to investigate the stability of different neural network architectures and develop memory efficient training algorithms. We also adopt the dynamical systems view of neural networks, but our aim is different from earlier works. We exploit the links between dynamical systems, optimal control, and neural networks to develop a novel distributed optimization algorithm. The proposed algorithm addresses the most significant obstacle for distributed algorithms for neural network optimization: the network weights cannot be updated until the forward propagation of the data, and backward propagation of the gradients are complete. Using the dynamical systems point of view, we interpret the layers of a (residual) neural network as the discretized dynamics of a dynamical system and exploit the relationship between the co-states (adjoints) of the optimal control problem and backpropagation. We then develop a parallel-in-time method that updates the parameters of the network without waiting for the forward or back propagation algorithms to complete in full. We establish the convergence of the proposed algorithm. Preliminary numerical results suggest that the algorithm is competitive and more efficient than the state-of-the-art.

References in corpus (5)

• Decoupled Parallel Backpropagation with Convergence Guarantee
• Distributed optimization of deeply nested systems
• Beyond Backprop: Online Alternating Minimization with Auxiliary Variables
• Global Convergence of Block Coordinate Descent in Deep Learning
• A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training

Cited by in corpus (4)

• The sharp, the flat and the shallow: Can weakly interacting agents learn to escape bad minima?
• Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework
• A Practical Layer-Parallel Training Algorithm for Residual Networks
• Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework