Weighted variation spaces and approximation by shallow ReLU networks

Abstract

We investigate the approximation of functions $f$ on a bounded domain $\Omega \subset {\mathbb{R}}^d$ by the outputs of single-hidden-layer ReLU neural networks of width $n$. This form of nonlinear $n$-term dictionary approximation has been intensely studied since it is the simplest case of neural network approximation (NNA). There are several celebrated approximation results for this form of NNA that introduce novel model classes of functions on $\Omega$ whose approximation rates do not grow unbounded with the input dimension. These novel classes include Barron classes, and classes based on sparsity or variation such as the Radon-domain BV classes. The present paper is concerned with the definition of these novel model classes on domains $\Omega$. The current definition of these model classes does not depend on the domain $\Omega$. A new and more proper definition of model classes on domains is given by introducing the concept of weighted variation spaces. These new model classes are intrinsic to the domain itself. The importance of these new model classes is that they are strictly larger than the classical (domain-independent) classes. Yet, it is shown that they maintain the same NNA rates.

References in corpus (10)

• Approximation and learning by greedy algorithms
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Sharp Bounds on the Approximation Rates, Metric Entropy, and $n$-widths of Shallow Neural Networks
• Sparsity of solutions for variational inverse problems with finite-dimensional data
• What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory
• Banach Space Representer Theorems for Neural Networks and Ridge Splines
• Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks
• Deep Learning Meets Sparse Regularization: A Signal Processing Perspective
• Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

Cited by in corpus (1)

• Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities