Mean-field Langevin System, Optimal Control and Deep Neural Networks

Abstract

In this paper, we study a regularised relaxed optimal control problem and, in particular, we are concerned with the case where the control variable is of large dimension. We introduce a system of mean-field Langevin equations, the invariant measure of which is shown to be the optimal control of the initial problem under mild conditions. Therefore, this system of processes can be viewed as a continuous-time numerical algorithm for computing the optimal control. As an application, this result endorses the solvability of the stochastic gradient descent algorithm for a wide class of deep neural networks.

arXiv comment25 pages

References in corpus (7)

• Neural Ordinary Differential Equations
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• On well-posedness of forward-backward SDEs-A unified approach
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
• Selection dynamics for deep neural networks

Cited by in corpus (7)

• Large Deviations and Exit-times for reflected McKean-Vlasov equations with self-stabilizing terms and superlinear drifts
• Mean-Field Neural ODEs via Relaxed Optimal Control
• Stochastic gradient descent and fast relaxation to thermodynamic equilibrium: a stochastic control approach
• Rough functional quantization and the support of McKean-Vlasov equations
• Ergodicity of the underdamped mean-field Langevin dynamics
• Second-Order Neural ODE Optimizer
• Dynamic Game Theoretic Neural Optimizer