How noise affects the Hessian spectrum in overparameterized neural networks
arXiv:1910.00195
Abstract
Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks. While some theoretical progress has been made, it still remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well. Here we show that for overparameterized networks with a degenerate valley in their loss landscape, SGD on average decreases the trace of the Hessian of the loss. We also generalize this result to other noise structures and show that isotropic noise in the non-degenerate subspace of the Hessian decreases its determinant. In addition to explaining SGDs role in sculpting the Hessian spectrum, this opens the door to new optimization approaches that may confer better generalization performance. We test our results with experiments on toy models and deep neural networks.
References in corpus (10)
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- Three Factors Influencing Minima in SGD
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Gradient Descent Happens in a Tiny Subspace
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
- Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
Cited by in corpus (7)
- Shape Matters: Understanding the Implicit Bias of the Noise Covariance
- Directional Pruning of Deep Neural Networks
- Generalization bounds for deep learning
- Is SGD a Bayesian sampler? Well, almost
- Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics
- Why flatness does and does not correlate with generalization for deep neural networks
- Subaging in underparametrized Deep Neural Networks