Nonlinear Approximation and (Deep) ReLU Networks

Abstract

This article is concerned with the approximation and expressive powers of deep neural networks. This is an active research area currently producing many interesting papers. The results most commonly found in the literature prove that neural networks approximate functions with classical smoothness to the same accuracy as classical linear methods of approximation, e.g. approximation by polynomials or by piecewise polynomials on prescribed partitions. However, approximation by neural networks depending on n parameters is a form of nonlinear approximation and as such should be compared with other nonlinear methods such as variable knot splines or n-term approximation from dictionaries. The performance of neural networks in targeted applications such as machine learning indicate that they actually possess even greater approximation power than these traditional methods of nonlinear approximation. The main results of this article prove that this is indeed the case. This is done by exhibiting large classes of functions which can be efficiently captured by neural networks where classical nonlinear methods fall short of the task. The present article purposefully limits itself to studying the approximation of univariate functions by ReLU networks. Many generalizations to functions of several variables and other activation functions can be envisioned. However, even in this simplest of settings considered here, a theory that completely quantifies the approximation power of neural networks is still lacking.

References in corpus (4)

• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Representation Benefits of Deep Feedforward Networks
• Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
• Quantified advantage of discontinuous weight selection in approximations with deep neural networks

Cited by in corpus (27)

• hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition
• Variational Physics-Informed Neural Networks For Solving Partial Differential Equations
• A Deep Learning approach to Reduced Order Modelling of Parameter Dependent Partial Differential Equations
• Gradient Dynamics of Shallow Univariate ReLU Networks
• Universal Approximation with Deep Narrow Networks
• On the capacity of deep generative networks for approximating distributions
• Neural Parametric Fokker-Planck Equations
• Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory
• Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization
• An error analysis of generative adversarial networks for learning distributions
• Sparse Subspace Clustering Friendly Deep Dictionary Learning for Hyperspectral Image Classification
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• Depth separation for reduced deep networks in nonlinear model reduction: Distilling shock waves in nonlinear hyperbolic problems
• Approximation of Smoothness Classes by Deep Rectifier Networks
• Some open questions on morphological operators and representations in the deep learning era
• Implicit Rugosity Regularization via Data Augmentation
• Solving PDEs on Unknown Manifolds with Machine Learning
• Partition of unity networks: deep hp-approximation
• Convergence of Deep ReLU Networks
• An Embedding of ReLU Networks and an Analysis of their Identifiability
• Hierarchically Compositional Tasks and Deep Convolutional Networks
• High-dimensional nonlinear approximation by parametric manifolds in Hölder-Nikol'skii spaces of mixed smoothness
• Depth-Adaptive Neural Networks from the Optimal Control viewpoint
• On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
• Convergence of Deep Convolutional Neural Networks
• Collocation approximation by deep neural ReLU networks for parametric elliptic PDEs with lognormal inputs
• Deep learning for inverse problems with unknown operator