An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks

Abstract

Deep learning is formulated as a discrete-time optimal control problem. This allows one to characterize necessary conditions for optimality and develop training algorithms that do not rely on gradients with respect to the trainable parameters. In particular, we introduce the discrete-time method of successive approximations (MSA), which is based on the Pontryagin's maximum principle, for training neural networks. A rigorous error estimate for the discrete MSA is obtained, which sheds light on its dynamics and the means to stabilize the algorithm. The developed methods are applied to train, in a rather principled way, neural networks with weights that are constrained to take values in a discrete set. We obtain competitive performance and interestingly, very sparse weights in the case of ternary networks, which may be useful in model deployment in low-memory devices.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Binarized Neural Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Learning may need only a few bits of synaptic precision
• BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights

Cited by in corpus (19)

• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• End-to-End Quantum Machine Learning Implemented with Controlled Quantum Dynamics
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Transport Analysis of Infinitely Deep Neural Network
• An Optimal Control Method to Compute the Most Likely Transition Path for Stochastic Dynamical Systems with Jumps
• Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
• Selection dynamics for deep neural networks
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• A Measure Theoretical Approach to the Mean-field Maximum Principle for Training NeurODEs
• Distributed Optimization for Over-Parameterized Learning
• An Optimal Control Approach to Early Stopping Variational Methods for Image Restoration
• DDPNOpt: Differential Dynamic Programming Neural Optimizer
• Amata: An Annealing Mechanism for Adversarial Training Acceleration
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Machine Learning and Control Theory
• Dynamic Game Theoretic Neural Optimizer
• Neural Delay Differential Equations