Provable approximation properties for deep neural networks

Abstract

We discuss approximation of functions using deep neural nets. Given a function $f$ on a $d$-dimensional manifold $\Gamma \subset {\mathbb{R}}^m$, we construct a sparsely-connected depth-4 neural network and bound its error in approximating $f$. The size of the network depends on dimension and curvature of the manifold $\Gamma$, the complexity of $f$, in terms of its wavelet description, and only weakly on the ambient dimension $m$. Essentially, our network computes wavelet functions, which are computed from Rectified Linear Units (ReLU)

accepted for publication in Applied and Computational Harmonic Analysis

References in corpus (8)

• Going Deeper with Convolutions
• Approximation and learning by greedy algorithms
• The Power of Depth for Feedforward Neural Networks
• An exact mapping between the Variational Renormalization Group and Deep Learning
• On the Computational Efficiency of Training Neural Networks
• A Probabilistic Theory of Deep Learning
• Deep nets for local manifold learning
• A mathematical motivation for complex-valued convolutional networks

Cited by in corpus (45)

• PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations
• The Modern Mathematics of Deep Learning
• MgNet: A Unified Framework of Multigrid and Convolutional Neural Network
• Are All Layers Created Equal?
• Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems
• Analysis of time-frequency scattering transforms
• Full error analysis for the training of deep neural networks
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Banach Space Representer Theorems for Neural Networks and Ridge Splines
• Approximation spaces of deep neural networks
• Automatic Processing and Solar Cell Detection in Photovoltaic Electroluminescence Images
• Random Vector Functional Link Networks for Function Approximation on Manifolds
• A Dimensionality Reduction Approach for Convolutional Neural Networks
• Efficient Representation of Low-Dimensional Manifolds using Deep Networks
• How degenerate is the parametrization of neural networks with the ReLU activation function?
• Universality of Deep Convolutional Neural Networks
• Approximation in shift-invariant spaces with deep ReLU neural networks
• Deep Radial Kernel Networks: Approximating Radially Symmetric Functions with Deep Networks
• Fast Calculation of Probabilistic Power Flow: A Model-based Deep Learning Approach
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• Learning-Based Reconstruction of FRI Signals
• Classification Logit Two-sample Testing by Neural Networks
• Chart Auto-Encoders for Manifold Structured Data
• Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theorem
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• Approximation of Functions on Manifolds in High Dimension from Noisy Scattered Data
• Variational Diffusion Autoencoders with Random Walk Sampling
• Enhanced Expressive Power and Fast Training of Neural Networks by Random Projections
• Approximation with Neural Networks in Variable Lebesgue Spaces
• Realization of spatial sparseness by deep ReLU nets with massive data
• Deep Neural Networks for Rotation-Invariance Approximation and Learning
• Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
• Construction of neural networks for realization of localized deep learning
• Machine learning based data retrieval for inverse scattering problems with incomplete data
• Some Approximation Bounds for Deep Networks
• Stability of the Decoupled Extended Kalman Filter Learning Algorithm in LSTM-Based Online Learning
• Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions
• Universal approximation properties of shallow quadratic neural networks
• Theory of Deep Convolutional Neural Networks II: Spherical Analysis
• Computational Analysis of Deformable Manifolds: from Geometric Modelling to Deep Learning
• Deep Regression for Repeated Measurements
• Generalization and Expressivity for Deep Nets