Selection dynamics for deep neural networks
arXiv:1905.09076
Abstract
This paper presents a partial differential equation framework for deep residual neural networks and for the associated learning problem. This is done by carrying out the continuum limits of neural networks with respect to width and depth. We study the wellposedness, the large time solution behavior, and the characterization of the steady states of the forward problem. Several useful time-uniform estimates and stability/instability conditions are presented. We state and prove optimality conditions for the inverse deep learning problem, using standard variational calculus, the Hamilton-Jacobi-Bellmann equation and the Pontryagin maximum principle. This serves to establish a mathematical foundation for investigating the algorithmic and theoretical connections between neural networks, PDE theory, variational analysis, optimal control, and deep learning.
arXiv comment27. arXiv admin note: text overlap with arXiv:1807.01083 by other authors
References in corpus (10)
- Searching for Activation Functions
- A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
- A Variational Perspective on Accelerated Methods in Optimization
- Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
- Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
- An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
- Deep Residual Learning and PDEs on Manifold
- Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions