Sharp Bounds on the Approximation Rates, Metric Entropy, and $n$-widths of Shallow Neural Networks

Abstract

In this article, we study approximation properties of the variation spaces corresponding to shallow neural networks with a variety of activation functions. We introduce two main tools for estimating the metric entropy, approximation rates, and $n$-widths of these spaces. First, we introduce the notion of a smoothly parameterized dictionary and give upper bounds on the non-linear approximation rates, metric entropy and $n$-widths of their absolute convex hull. The upper bounds depend upon the order of smoothness of the parameterization. This result is applied to dictionaries of ridge functions corresponding to shallow neural networks, and they improve upon existing results in many cases. Next, we provide a method for lower bounding the metric entropy and $n$-widths of variation spaces which contain certain classes of ridge functions. This result gives sharp lower bounds on the $L^2$-approximation rates, metric entropy, and $n$-widths for variation spaces corresponding to neural networks with a range of important activation functions, including ReLU${}^k$ activation functions and sigmoidal activation functions with bounded variation.

has been merged with arXiv:2106.14997

References in corpus (3)

• Approximation and learning by greedy algorithms
• The Finite Neuron Method and Convergence Analysis
• What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory

Cited by in corpus (14)

• Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks
• Deep Learning Meets Sparse Regularization: A Signal Processing Perspective
• Optimal rates of approximation by shallow ReLU${}^k$ neural networks and applications to nonparametric regression
• Entropy-based convergence rates of greedy algorithms
• A reduced conjugate gradient basis method for fractional diffusion
• On the optimality of target-data-dependent kernel greedy interpolation in Sobolev Reproducing Kernel Hilbert Spaces
• Spectral Barron space for deep neural network approximation
• A neuron-wise subspace correction method for the finite neuron method
• Weighted variation spaces and approximation by shallow ReLU networks
• Inequalities between s-numbers
• A new analysis of empirical interpolation methods and Chebyshev greedy algorithms
• Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities
• Embedding Inequalities for Barron-type Spaces
• An iterative method for the solution of Laplace-like equations in high and very high space dimensions