Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective
arXiv:2001.06931
Abstract
It is known that any target function is realized in a sufficiently small neighborhood of any randomly connected deep network, provided the width (the number of neurons in a layer) is sufficiently large. There are sophisticated theories and discussions concerning this striking fact, but rigorous theories are very complicated. We give an elementary geometrical proof by using a simple model for the purpose of elucidating its structure. We show that high-dimensional geometry plays a magical role: When we project a high-dimensional sphere of radius 1 to a low-dimensional subspace, the uniform distribution over the sphere reduces to a Gaussian distribution of negligibly small covariances.
References in corpus (11)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- On Exact Computation with an Infinitely Wide Neural Net
- Exponential expressivity in deep neural networks through transient chaos
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
- Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
- Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
- Fisher Information and Natural Gradient Learning of Random Deep Networks
- Approximation power of random neural networks
- State Concentration Exponent as a Measure of Quickness in Kauffman-type Networks
- The global optimum of shallow neural network is attained by ridgelet transform