Explorations on high dimensional landscapes
arXiv:1412.6615
Abstract
Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science. We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points. This is in contrast with the low dimensional picture in which this band is wide. Our simulations agree with the previous theoretical work on spin glasses that proves the existence of such a band when the dimension of the domain tends to infinity. Furthermore our experiments on teacher-student networks with the MNIST dataset establish a similar phenomenon in deep networks. We finally observe that both the gradient descent and the stochastic gradient descent methods can reach this level within the same number of steps.
11 pages, 8 figures, workshop contribution at ICLR 2015
References in corpus (6)
- Theano: new features and speed improvements
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Extreme Value Statistics of Eigenvalues of Gaussian Random Matrices
- Free energy and complexity of spherical bipartite models
- High-Dimensional Random Fields and Random Matrix Theory
- A walk in the statistical mechanical formulation of neural networks
Cited by in corpus (11)
- AdaNet: Adaptive Structural Learning of Artificial Neural Networks
- Local minima in training of neural networks
- The Loss Surfaces of Neural Networks with General Activation Functions
- Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
- Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
- Deep Learning applied to Road Traffic Speed forecasting
- Optimizing Mode Connectivity via Neuron Alignment
- Universal characteristics of deep neural network loss surfaces from random matrix theory
- Appearance of Random Matrix Theory in Deep Learning
- Error Estimation and Correction from within Neural Network Differential Equation Solvers
- Algebraically-Informed Deep Networks (AIDN): A Deep Learning Approach to Represent Algebraic Structures