A Mean-Field Optimal Control Formulation of Deep Learning

Abstract

Recent work linking deep neural networks and dynamical systems opened up new avenues to analyze deep learning. In particular, it is observed that new insights can be obtained by recasting deep learning as an optimal control problem on difference or differential equations. However, the mathematical aspects of such a formulation have not been systematically explored. This paper introduces the mathematical formulation of the population risk minimization problem in deep learning as a mean-field optimal control problem. Mirroring the development of classical optimal control, we state and prove optimality conditions of both the Hamilton-Jacobi-Bellman type and the Pontryagin type. These mean-field results reflect the probabilistic nature of the learning problem. In addition, by appealing to the mean-field Pontryagin's maximum principle, we establish some quantitative relationships between population and empirical learning problems. This serves to establish a mathematical foundation for investigating the algorithmic and theoretical connections between optimal control and deep learning.

arXiv comment44 pages

References in corpus (2)

• Stable Architectures for Deep Neural Networks
• Mean-Field Optimal Control

Cited by in corpus (17)

• A Review on Deep Learning in Medical Image Reconstruction
• Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning
• The Random Feature Model for Input-Output Maps between Banach Spaces
• Machine Learning from a Continuous Viewpoint
• Neural network architectures using min-plus algebra for solving certain high dimensional optimal control problems and Hamilton-Jacobi PDEs
• Operator Learning Using Random Features: A Tool for Scientific Computing
• Semiconcavity and Sensitivity Analysis in Mean-Field Optimal Control and Applications
• Computational Mean-field Games on Manifolds
• Random Features for High-Dimensional Nonlocal Mean-Field Games
• From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks
• Optimal Dirichlet Boundary Control by Fourier Neural Operators Applied to Nonlinear Optics
• Framework for Learning and Control in the Classical and Quantum Domains
• Disordered high-dimensional optimal control
• Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games
• Optimal control of diffusion processes: $\infty$-order variational analysis and numerical solution
• Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
• A Mean Field Games Perspective on Evolutionary Clustering