How Powerful are Shallow Neural Networks with Bandlimited Random Weights?
arXiv:2008.08427
Abstract
We investigate the expressive power of depth-2 bandlimited random neural networks. A random net is a neural network where the hidden layer parameters are frozen with random assignment, and only the output layer parameters are trained by loss minimization. Using random weights for a hidden layer is an effective method to avoid non-convex optimization in standard gradient descent learning. It has also been adopted in recent deep learning theories. Despite the well-known fact that a neural network is a universal approximator, in this study, we mathematically show that when hidden parameters are distributed in a bounded domain, the network may not achieve zero approximation error. In particular, we derive a new nontrivial approximation error lower bound. The proof utilizes the technique of ridgelet analysis, a harmonic analysis method designed for neural networks. This method is inspired by fundamental principles in classical signal processing, specifically the idea that signals with limited bandwidth may not always be able to perfectly recreate the original signal. We corroborate our theoretical results with various simulation studies, and generally, two main take-home messages are offered: (i) Not any distribution for selecting random weights is feasible to build a universal approximator; (ii) A suitable assignment of random weights exists but to some degree is associated with the complexity of the target function.
arXiv commentPublished as a conference paper at ICML 2023
References in corpus (16)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Stochastic Configuration Networks: Fundamentals and Algorithms
- On Lazy Training in Differentiable Programming
- Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
- Approximation with Random Bases: Pro et Contra
- A Priori Estimates of the Population Risk for Two-layer Neural Networks
- Proving the Lottery Ticket Hypothesis: Pruning is All You Need
- SGD Learns the Conjugate Kernel Class of the Network
- On the ability of neural nets to express distributions
- Banach Space Representer Theorems for Neural Networks and Ridge Splines
- Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
- Approximation power of random neural networks
- Emerging applications of geometric multiscale analysis
- On Random Weights for Texture Generation in One Layer Neural Networks
- Fast Approximation and Estimation Bounds of Kernel Quadrature for Infinitely Wide Models