Mean-Field Neural ODEs via Relaxed Optimal Control

Abstract

We develop a framework for the analysis of deep neural networks and neural ODE models that are trained with stochastic gradient algorithms. We do that by identifying the connections between control theory, deep learning and theory of statistical sampling. We derive Pontryagin's optimality principle and study the corresponding gradient flow in the form of Mean-Field Langevin dynamics (MFLD) for solving relaxed data-driven control problems. Subsequently, we study uniform-in-time propagation of chaos of time-discretised MFLD. We derive explicit convergence rate in terms of the learning rate, the number of particles/model parameters and the number of iterations of the gradient algorithm. In addition, we study the error arising when using a finite training data set and thus provide quantitive bounds on the generalisation error. Crucially, the obtained rates are dimension-independent. This is possible by exploiting the regularity of the model with respect to the measure over the parameter space.

References in corpus (22)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Understanding Deep Convolutional Networks
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• From the master equation to mean field game limit theory: A central limit theorem
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
• Geometry of Optimization and Implicit Regularization in Deep Learning
• When Do Neural Networks Outperform Kernel Methods?
• Augmented Neural ODEs
• Maximum Mean Discrepancy Gradient Flow
• Weak quantitative propagation of chaos via differential calculus on the space of measures
• Statistical Inference for Generative Models with Maximum Mean Discrepancy
• Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
• Mean-field Langevin System, Optimal Control and Deep Neural Networks
• On finite-difference approximations for normalized Bellman equations
• Rate of propagation of chaos for diffusive stochastic particle systems via Girsanov transformation
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer

Cited by in corpus (5)

• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• STEER: Simple Temporal Regularization For Neural ODEs
• Semiconcavity and Sensitivity Analysis in Mean-Field Optimal Control and Applications
• Quantitative Propagation of Chaos for SGD in Wide Neural Networks
• Ergodicity of the underdamped mean-field Langevin dynamics