Deep Network Approximation for Smooth Functions

Abstract

This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width $\mathcal{O}(N)$ and depth $\mathcal{O}(L)$ with an approximation error $\mathcal{O}(N^{-L})$. Through local Taylor expansions and their deep ReLU network approximations, we show that deep ReLU networks of width $\mathcal{O}(N\ln N)$ and depth $\mathcal{O}(L\ln L)$ can approximate $f\in C^s([0,1{]}^d)$ with a nearly optimal approximation error $\mathcal{O}(\Vert f{\Vert }_{C^s([0,1{]}^d)}{N}^{-2s/d}{L}^{-2s/d})$. Our estimate is non-asymptotic in the sense that it is valid for arbitrary width and depth specified by $N\in {\mathbb{N}}^{+}$ and $L\in {\mathbb{N}}^{+}$, respectively.

References in corpus (1)

• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks

Cited by in corpus (39)

• Deep Potentials for Materials Science
• Deep Nitsche Method: Deep Ritz Method with Essential Boundary Conditions
• Neural Network Approximation: Three Hidden Layers Are Enough
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• Evolutional Deep Neural Network
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• Efficient approximation of high-dimensional functions with neural networks
• ReLU Deep Neural Networks from the Hierarchical Basis Perspective
• Symplectic Learning for Hamiltonian Neural Networks
• On the capacity of deep generative networks for approximating distributions
• Factor Augmented Sparse Throughput Deep ReLU Neural Networks for High Dimensional Regression
• Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularities
• Approximation bounds for norm constrained neural networks with applications to regression and GANs
• Structure Probing Neural Network Deflation
• An error analysis of generative adversarial networks for learning distributions
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• On Representing (Anti)Symmetric Functions
• Learning-informed parameter identification in nonlinear time-dependent PDEs
• On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
• Approximation in shift-invariant spaces with deep ReLU neural networks
• The gap between theory and practice in function approximation with deep neural networks
• Deep Nonparametric Regression on Approximate Manifolds: Non-Asymptotic Error Bounds with Polynomial Prefactors
• On Deep Instrumental Variables Estimate
• Reproducing Activation Function for Deep Learning
• Power Series Expansion Neural Network
• Solving PDEs on Unknown Manifolds with Machine Learning
• Discretization of parameter identification in PDEs using Neural Networks
• Deep Neural Networks Are Effective At Learning High-Dimensional Hilbert-Valued Functions From Limited Data
• Size and Depth Separation in Approximating Benign Functions with Neural Networks
• The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation
• Calibrating multi-dimensional complex ODE from noisy data via deep neural networks
• Two neural-network-based methods for solving obstacle problems
• Adaptive Learning on the Grids for Elliptic Hemivariational Inequalities
• Multiscale and Nonlocal Learning for PDEs using Densely Connected RNNs
• Strong overall error analysis for the training of artificial neural networks via random initializations
• Sparsity-Probe: Analysis tool for Deep Learning Models
• Stationary Density Estimation of Itô Diffusions Using Deep Learning