The Loss Surfaces of Neural Networks with General Activation Functions

Abstract

The loss surfaces of deep neural networks have been the subject of several studies, theoretical and experimental, over the last few years. One strand of work considers the complexity, in the sense of local optima, of high dimensional random functions with the aim of informing how local optimisation methods may perform in such complicated settings. Prior work of Choromanska et al (2015) established a direct link between the training loss surfaces of deep multi-layer perceptron networks and spherical multi-spin glass models under some very strong assumptions on the network and its data. In this work, we test the validity of this approach by removing the undesirable restriction to ReLU activation functions. In doing so, we chart a new path through the spin glass complexity calculations using supersymmetric methods in Random Matrix Theory which may prove useful in other contexts. Our results shed new light on both the strengths and the weaknesses of spin glass models in this context.

arXiv comment50 pages, 11 figures; references added for Kac-Rice reduction to RMT method; updates following JSTAT review and publication

References in corpus (17)

• Decoupled Weight Decay Regularization
• On Calibration of Modern Neural Networks
• The Loss Surfaces of Multilayer Networks
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• How many eigenvalues of a Gaussian random matrix are positive?
• Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitions
• Rethinking mean-field glassy dynamics and its relation with the energy landscape: the awkward case of the spherical mixed p-spin model
• Comparing Dynamics: Deep Neural Networks versus Glassy Systems
• Quenched complexity of the p-spin spherical spin-glass with external magnetic field
• A shortcut through the Coulomb gas method for spectral linear statistics on random matrices
• Characteristic polynomials of random Hermitian matrices and Duistermaat-Heckman localisation on non-compact Kaehler manifolds
• Explorations on high dimensional landscapes
• Supersymmetric quenched complexity in the Sherrington-Kirkpatrick model
• Complexity of the Sherrington-Kirkpatrick Model in the Annealed Approximation
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Optimal Piecewise Linear Function Approximation for GPU-based Applications
• On Random Matrix Averages Involving Half-Integer Powers of GOE Characteristic Polynomials

Cited by in corpus (9)

• Exponential growth of random determinants beyond invariance
• Counting equilibria in a random non-gradient dynamics with heterogeneous relaxation rates
• Nonlinearity-generated Resilience in Large Complex Systems
• A spin-glass model for the loss surfaces of generative adversarial networks
• Local convexity of the TAP free energy and AMP convergence for Z2-synchronization
• Universal characteristics of deep neural network loss surfaces from random matrix theory
• Superposition of Random Plane Waves in High Spatial Dimensions: Random Matrix Approach to Landscape Complexity
• Appearance of Random Matrix Theory in Deep Learning
• Optimization landscape in the simplest constrained random least-square problem