An Investigation into Neural Net Optimization via Hessian Eigenvalue Density

Abstract

To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly analyze a crucial structural feature of the spectra: in non-batch normalized networks, we observe the rapid appearance of large isolated eigenvalues in the spectrum, along with a surprising concentration of the gradient in the corresponding eigenspaces. In batch normalized networks, these two effects are almost absent. We characterize these effects, and explain how they affect optimization speed through both theory and experiments. As part of this work, we adapt advanced tools from numerical linear algebra that allow scalable and accurate estimation of the entire Hessian spectrum of ImageNet-scale neural networks; this technique may be of independent interest in other applications.

21 pages, 19 figures

Cited by in corpus (59)

• Scaling Laws for Neural Language Models
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• When Do Neural Networks Outperform Kernel Methods?
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Stiffness: A New Perspective on Generalization in Neural Networks
• The Early Phase of Neural Network Training
• Lipschitz Recurrent Neural Networks
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• PyHessian: Neural Networks Through the Lens of the Hessian
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Mathematical Models of Overparameterized Neural Networks
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Deep Learning Through the Lens of Example Difficulty
• The Loss Surfaces of Neural Networks with General Activation Functions
• Emergent properties of the local geometry of neural loss landscapes
• Hessian-based toolbox for reliable and interpretable machine learning in physics
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• Directional Pruning of Deep Neural Networks
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
• How noise affects the Hessian spectrum in overparameterized neural networks
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Average-case Acceleration Through Spectral Density Estimation
• Neural Anisotropy Directions
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Analysis of stochastic Lanczos quadrature for spectrum approximation
• A Loss Curvature Perspective on Training Instability in Deep Learning
• The asymptotic spectrum of the Hessian of DNN throughout training
• AdaSGD: Bridging the gap between SGD and Adam
• Quasi-Newton's method in the class gradient defined high-curvature subspace
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
• Optimal Sketching for Trace Estimation
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• Sparse Flows: Pruning Continuous-depth Models
• Learning with Gradient Descent and Weakly Convex Losses
• SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
• De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors
• Categorical semantics of a simple differential programming language
• Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
• Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
• Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
• Low Rank Saddle Free Newton: A Scalable Method for Stochastic Nonconvex Optimization
• A Dynamic Sampling Adaptive-SGD Method for Machine Learning
• Federated Few-Shot Learning with Adversarial Learning
• Appearance of Random Matrix Theory in Deep Learning
• Structured Directional Pruning via Perturbation Orthogonal Projection
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II
• Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models
• Group Whitening: Balancing Learning Efficiency and Representational Capacity
• An Investigation into the Stochasticity of Batch Whitening