The global optimum of shallow neural network is attained by ridgelet transform
arXiv:1805.07517
Abstract
We prove that the global minimum of the backpropagation (BP) training problem of neural networks with an arbitrary nonlinear activation is given by the ridgelet transform. A series of computational experiments show that there exists an interesting similarity between the scatter plot of hidden parameters in a shallow neural network after the BP training and the spectrum of the ridgelet transform. By introducing a continuous model of neural networks, we reduce the training problem to a convex optimization in an infinite dimensional Hilbert space, and obtain the explicit expression of the global optimizer via the ridgelet transform.
arXiv commentunder review
References in corpus (8)
- WaveNet: A Generative Model for Raw Audio
- The Loss Surfaces of Multilayer Networks
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Essentially No Barriers in Neural Network Energy Landscape
- The loss surface of deep and wide neural networks
- No bad local minima: Data independent training error guarantees for multilayer neural networks
- Depth Creates No Bad Local Minima
Cited by in corpus (4)
- The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks
- Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective
- Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum
- Fast Approximation and Estimation Bounds of Kernel Quadrature for Infinitely Wide Models