Emergent properties of the local geometry of neural loss landscapes

Abstract

The local geometry of high dimensional neural network loss landscapes can both challenge our cherished theoretical intuitions as well as dramatically impact the practical success of neural network training. Indeed recent works have observed 4 striking local properties of neural loss landscapes on classification tasks: (1) the landscape exhibits exactly $C$ directions of high positive curvature, where $C$ is the number of classes; (2) gradient directions are largely confined to this extremely low dimensional subspace of positive Hessian curvature, leaving the vast majority of directions in weight space unexplored; (3) gradient descent transiently explores intermediate regions of higher positive curvature before eventually finding flatter minima; (4) training can be successful even when confined to low dimensional {\it random} affine hyperplanes, as long as these hyperplanes intersect a Goldilocks zone of higher than average curvature. We develop a simple theoretical model of gradients and Hessians, justified by numerical experiments on architectures and datasets used in practice, that {\it simultaneously} accounts for all $4$ of these surprising and seemingly unrelated properties. Our unified model provides conceptual insights into the emergence of these properties and makes connections with diverse topics in neural networks, random matrix theory, and spin glasses, including the neural tangent kernel, BBP phase transitions, and Derrida's random energy model.

10 pages, 8 figures

References in corpus (2)

• Gradient Descent Happens in a Tiny Subspace
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density

Cited by in corpus (13)

• Consensus Control for Decentralized Deep Learning
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Linear Mode Connectivity in Multitask and Continual Learning
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Taxonomizing local versus global structure in neural network loss landscapes
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Improving Neural Network Training in Low Dimensional Random Bases
• Quasi-Newton's method in the class gradient defined high-curvature subspace
• Does the Data Induce Capacity Control in Deep Learning?
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• On the Impact of Stable Ranks in Deep Nets