Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory

Abstract

The problem of solving partial differential equations (PDEs) can be formulated into a least-squares minimization problem, where neural networks are used to parametrize PDE solutions. A global minimizer corresponds to a neural network that solves the given PDE. In this paper, we show that the gradient descent method can identify a global minimizer of the least-squares optimization for solving second-order linear PDEs with two-layer neural networks under the assumption of over-parametrization. We also analyze the generalization error of the least-squares optimization for second-order linear PDEs and two-layer neural networks, when the right-hand-side function of the PDE is in a Barron-type space and the least-squares optimization is regularized with a Barron-type norm, without the over-parametrization assumption.

References in corpus (26)

• DGM: A deep learning algorithm for solving partial differential equations
• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• Densely Connected Convolutional Networks
• Solving high-dimensional partial differential equations using deep learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• A unified deep artificial neural network approach to partial differential equations in complex geometries
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Deep Network Approximation for Smooth Functions
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Convergence of the Deep BSDE Method for Coupled FBSDEs
• Deep Nitsche Method: Deep Ritz Method with Essential Boundary Conditions
• Neural Network Approximation: Three Hidden Layers Are Enough
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Nonlinear Approximation via Compositions
• Int-Deep: A Deep Learning Initialized Iterative Method for Nonlinear Problems
• SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• The phase diagram of approximation rates for deep neural networks
• A Priori Estimates of the Population Risk for Residual Networks
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
• Approximation in shift-invariant spaces with deep ReLU neural networks

Cited by in corpus (24)

• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Tackling the Curse of Dimensionality with Physics-Informed Neural Networks
• Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology
• When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
• Neural Network Approximation: Three Hidden Layers Are Enough
• An overview on deep learning-based approximation methods for partial differential equations
• Evolutional Deep Neural Network
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Convergence Rate Analysis for Deep Ritz Method
• A rate of convergence of Physics Informed Neural Networks for the linear second order elliptic PDEs
• Wasserstein Generative Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• Neural Networks Enforcing Physical Symmetries in Nonlinear Dynamical Lattices: The Case Example of the Ablowitz-Ladik Model
• A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equations
• Tackling the Curse of Dimensionality in Fractional and Tempered Fractional PDEs with Physics-Informed Neural Networks
• Structure Probing Neural Network Deflation
• Imaging Conductivity from Current Density Magnitude using Neural Networks
• A Priori Analysis of Stable Neural Network Solutions to Numerical PDEs
• Reproducing Activation Function for Deep Learning
• Solving PDEs on Unknown Manifolds with Machine Learning
• On the Representation of Solutions to Elliptic PDEs in Barron Spaces
• The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation
• Analysis of Deep Ritz Methods for Laplace Equations with Dirichlet Boundary Conditions
• Approximation of Functionals by Neural Network without Curse of Dimensionality
• Stationary Density Estimation of Itô Diffusions Using Deep Learning