Machine Learning from a Continuous Viewpoint

Abstract

We present a continuous formulation of machine learning, as a problem in the calculus of variations and differential-integral equations, in the spirit of classical numerical analysis. We demonstrate that conventional machine learning models and algorithms, such as the random feature model, the two-layer neural network model and the residual neural network model, can all be recovered (in a scaled form) as particular discretizations of different continuous formulations. We also present examples of new models, such as the flow-based random feature model, and new algorithms, such as the smoothed particle method and spectral method, that arise naturally from this continuous formulation. We discuss how the issues of generalization error and implicit regularization can be studied under this framework.

arXiv commentpublished version

References in corpus (13)

• DGM: A deep learning algorithm for solving partial differential equations
• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• Solving high-dimensional partial differential equations using deep learning
• Reconciling modern machine learning practice and the bias-variance trade-off
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Ab-Initio Solution of the Many-Electron Schrödinger Equation with Deep Neural Networks
• Stable Architectures for Deep Neural Networks
• Neural Network with Unbounded Activation Functions is Universal Approximator
• Solving Many-Electron Schrödinger Equation Using Deep Neural Networks
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• A Mean-Field Optimal Control Formulation of Deep Learning

Cited by in corpus (14)

• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning
• SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs
• Theory of the Frequency Principle for General Deep Neural Networks
• Model-free tracking control of complex dynamical trajectories with machine learning
• Accurate Deep Potential model for the Al-Cu-Mg alloy in the full concentration space
• Stochastic asymptotical regularization for linear inverse problems
• Operator Learning Using Random Features: A Tool for Scientific Computing
• Generalization Error of GAN from the Discriminator's Perspective
• Learning Domain-Independent Green's Function For Elliptic Partial Differential Equations
• A Mathematical Framework for Learning Probability Distributions
• Data-informed Deep Optimization
• Efficient Learning of Long-Range and Equivariant Quantum Systems
• An efficient stochastic particle method for high-dimensional nonlinear PDEs