High-dimensional approximation spaces of artificial neural networks and applications to partial differential equations

Abstract

In this paper we develop a new machinery to study the capacity of artificial neural networks (ANNs) to approximate high-dimensional functions without suffering from the curse of dimensionality. Specifically, we introduce a concept which we refer to as approximation spaces of artificial neural networks and we present several tools to handle those spaces. Roughly speaking, approximation spaces consist of sequences of functions which can, in a suitable way, be approximated by ANNs without curse of dimensionality in the sense that the number of required ANN parameters to approximate a function of the sequence with an accuracy $\epsilon >0$ grows at most polynomially both in the reciprocal $1/\epsilon$ of the required accuracy and in the dimension $d\in \mathbb{N}=\{1,2,3,\dots \}$ of the function. We show that these approximation spaces are closed under various operations including linear combinations, formations of limits, and infinite compositions. To illustrate the utility of the machinery proposed in this paper, we employ the developed theory to prove that ANNs have the capacity to overcome the curse of dimensionality in the numerical approximation of certain first order transport partial differential equations (PDEs). We even prove that approximation spaces are closed under flows of first order transport PDEs.

31 pages

References in corpus (16)

• DGM: A deep learning algorithm for solving partial differential equations
• Solving high-dimensional partial differential equations using deep learning
• Recent Trends in Deep Learning Based Natural Language Processing
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Loss of regularity for Kolmogorov equations
• Strong error analysis for stochastic gradient descent optimization algorithms
• Full error analysis for the training of deep neural networks
• Efficient approximation of high-dimensional functions with neural networks
• Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems
• Uniform error estimates for artificial neural network approximations for heat equations
• Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning rates
• Overcoming the curse of dimensionality in the numerical approximation of high-dimensional semilinear elliptic partial differential equations
• Stochastic modified equations and adaptive stochastic gradient algorithms
• On the Alekseev-Gröbner formula in Banach spaces

Cited by in corpus (1)

• An overview on deep learning-based approximation methods for partial differential equations