Deep Network Approximation Characterized by Number of Neurons

Abstract

This paper quantitatively characterizes the approximation power of deep feed-forward neural networks (FNNs) in terms of the number of neurons. It is shown by construction that ReLU FNNs with width $\mathcal{O}(\max \{d\lfloor N^{1/d}\rfloor ,N+1\})$ and depth $\mathcal{O}(L)$ can approximate an arbitrary Hölder continuous function of order $\alpha \in (0,1]$ on $[0,1{]}^d$ with a nearly tight approximation rate $\mathcal{O}\left(\sqrt{d}{N}^{-2\alpha /d}{L}^{-2\alpha /d}\right)$ measured in $L^p$-norm for any $N,L\in {\mathbb{N}}^{+}$ and $p\in [1,\infty ]$. More generally for an arbitrary continuous function $f$ on $[0,1{]}^d$ with a modulus of continuity ${\omega }_f(\cdot )$, the constructive approximation rate is $\mathcal{O}(\sqrt{d}{\omega }_f(N^{-2/d}{L}^{-2/d}))$. We also extend our analysis to $f$ on irregular domains or those localized in an $\epsilon$-neighborhood of a $d_{\mathcal{M}}$-dimensional smooth manifold $\mathcal{M}\subseteq [0,1{]}^d$ with $d_{\mathcal{M}}\ll d$. Especially, in the case of an essentially low-dimensional domain, we show an approximation rate $\mathcal{O}({\omega }_f(\frac{\epsilon }{1-\delta }\sqrt{\frac{d}{d_{\delta }}}+\epsilon )+\sqrt{d}{\omega }_f(\frac{\sqrt{d}}{(1-\delta )\sqrt{d_{\delta }}}{N}^{-2/d_{\delta }}{L}^{-2/d_{\delta }}))$ for ReLU FNNs to approximate $f$ in the $\epsilon$-neighborhood, where $d_{\delta }=\mathcal{O}\left(d_{\mathcal{M}}\frac{\ln (d/\delta )}{{\delta }^2}\right)$ for any $\delta \in (0,1)$ as a relative error for a projection to approximate an isometry when projecting $\mathcal{M}$ to a $d_{\delta }$-dimensional domain.

Cited by in corpus (20)

• Deep Network Approximation for Smooth Functions
• Neural Network Approximation: Three Hidden Layers Are Enough
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Full error analysis for the training of deep neural networks
• On the capacity of deep generative networks for approximating distributions
• Optimal rates of approximation by shallow ReLU${}^k$ neural networks and applications to nonparametric regression
• Approximation bounds for norm constrained neural networks with applications to regression and GANs
• Approximation in shift-invariant spaces with deep ReLU neural networks
• Classification Logit Two-sample Testing by Neural Networks
• Least-Squares Neural Network (LSNN) Method For Linear Advection-Reaction Equation: Discontinuity Interface
• An operator learning perspective on parameter-to-observable maps
• Deep ReLU networks and high-order finite element methods II: Chebyshev emulation
• Why Shallow Networks Struggle to Approximate and Learn High Frequencies
• A Natural Deep Ritz Method for Essential Boundary Value Problems
• Don't Fear Peculiar Activation Functions: EUAF and Beyond
• On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks
• Strong overall error analysis for the training of artificial neural networks via random initializations
• Deep Regression for Repeated Measurements
• Structured and Balanced Multi-Component and Multi-Layer Neural Networks