Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularities

Abstract

We prove exponential expressivity with stable ReLU Neural Networks (ReLU NNs) in $H^1(\Omega )$ for weighted analytic function classes in certain polytopal domains $\Omega$, in space dimension $d=2,3$. Functions in these classes are locally analytic on open subdomains $D\subset \Omega$, but may exhibit isolated point singularities in the interior of $\Omega$ or corner and edge singularities at the boundary $\partial \Omega$. The exponential expression rate bounds proved here imply uniform exponential expressivity by ReLU NNs of solution families for several elliptic boundary and eigenvalue problems with analytic data. The exponential approximation rates are shown to hold in space dimension $d=2$ on Lipschitz polygons with straight sides, and in space dimension $d=3$ on Fichera-type polyhedral domains with plane faces. The constructive proofs indicate in particular that NN depth and size increase poly-logarithmically with respect to the target NN approximation accuracy $\epsilon >0$ in $H^1(\Omega )$. The results cover in particular solution sets of linear, second order elliptic PDEs with analytic data and certain nonlinear elliptic eigenvalue problems with analytic nonlinearities and singular, weighted analytic potentials as arise in electron structure models. In the latter case, the functions correspond to electron densities that exhibit isolated point singularities at the positions of the nuclei. Our findings provide in particular mathematical foundation of recently reported, successful uses of deep neural networks in variational electron structure algorithms.

arXiv commentFound Comput Math (2022)

References in corpus (20)

• DeepXDE: A deep learning library for solving differential equations
• DGM: A deep learning algorithm for solving partial differential equations
• Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
• Deep neural network solution of the electronic Schrödinger equation
• Ab-Initio Solution of the Many-Electron Schrödinger Equation with Deep Neural Networks
• Solving Many-Electron Schrödinger Equation Using Deep Neural Networks
• Deep Network Approximation for Smooth Functions
• ReLU Deep Neural Networks and Linear Finite Elements
• Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations
• PFNN: A Penalty-Free Neural Network Method for Solving a Class of Second-Order Boundary-Value Problems on Complex Geometries
• Optimal approximation of continuous functions by very deep ReLU networks
• Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
• DNN Expression Rate Analysis of High-dimensional PDEs: Application to Option Pricing
• Analytic Regularity for Linear Elliptic Systems in Polygons and Polyhedra
• A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients
• Error estimates of residual minimization using neural networks for linear PDEs
• Analytic structure of solutions to multiconfiguration equations
• Regularity and $hp$ discontinuous Galerkin finite element approximation of linear elliptic eigenvalue problems with singular potentials
• Analytic regularity for the incompressible Navier-Stokes equations in polygons
• Weighted analyticity of Hartree-Fock eigenfunctions

Cited by in corpus (6)

• Deep Learning in Deterministic Computational Mechanics
• The Modern Mathematics of Deep Learning
• ReLU Deep Neural Networks from the Hierarchical Basis Perspective
• Exponential Convergence of Deep Operator Networks for Elliptic Partial Differential Equations
• Deep ReLU networks and high-order finite element methods II: Chebyshev emulation
• A Low Rank Neural Representation of Entropy Solutions