Geometry of energy landscapes and the optimizability of deep neural networks

Abstract

Deep neural networks are workhorse models in machine learning with multiple layers of non-linear functions composed in series. Their loss function is highly non-convex, yet empirically even gradient descent minimisation is sufficient to arrive at accurate and predictive models. It is hitherto unknown why are deep neural networks easily optimizable. We analyze the energy landscape of a spin glass model of deep neural networks using random matrix theory and algebraic geometry. We analytically show that the multilayered structure holds the key to optimizability: Fixing the number of parameters and increasing network depth, the number of stationary points in the loss function decreases, minima become more clustered in parameter space, and the tradeoff between the depth and width of minima becomes less severe. Our analytical results are numerically verified through comparison with neural networks trained on a set of classical benchmark datasets. Our model uncovers generic design principles of machine learning models.

References in corpus (16)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Universally Sloppy Parameter Sensitivities in Systems Biology
• Ab initio Random Structure Searching
• Why does deep and cheap learning work so well?
• The Number of Eigenvalues of a Tensor
• Thermodynamics and the Global Optimization of Lennard-Jones clusters
• Some Further Results for the Stationary Points and Dynamics of Supercooled Liquids
• The statistics of critical points of Gaussian fields on large-dimensional spaces
• Complexity of random smooth functions on the high-dimensional sphere
• Perspective: Energy Landscapes for Machine Learning
• Characterizing the network topology of the energy landscapes of atomic clusters
• Turning intractable counting into sampling: computing the configurational entropy of three-dimensional jammed packings
• Power-law distributions for the areas of the basins of attraction on a potential energy landscape
• Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
• Structural analysis of high-dimensional basins of attraction
• The Loss Surface of XOR Artificial Neural Networks

Cited by in corpus (8)

• Machine learning with neural networks
• Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
• Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
• Is SGD a Bayesian sampler? Well, almost
• On-Site Potential Creates Complexity in Systems with Disordered Coupling
• Data-driven effective model shows a liquid-like deep learning
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• Generalisation in fully-connected neural networks for time series forecasting