Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
arXiv:1907.10732
Abstract
While stochastic gradient descent (SGD) and variants have been surprisingly successful for training deep nets, several aspects of the optimization dynamics and generalization are still not well understood. In this paper, we present new empirical observations and theoretical results on both the optimization dynamics and generalization behavior of SGD for deep nets based on the Hessian of the training loss and associated quantities. We consider three specific research questions: (1) what is the relationship between the Hessian of the loss and the second moment of stochastic gradients (SGs)? (2) how can we characterize the stochastic optimization dynamics of SGD with fixed and adaptive step sizes and diagonal pre-conditioning based on the first and second moments of SGs? and (3) how can we characterize a scale-invariant generalization bound of deep nets based on the Hessian of the loss, which by itself is not scale invariant? We shed light on these three questions using theoretical results supported by extensive empirical observations, with experiments on synthetic data, MNIST, and CIFAR-10, with different batch sizes, and with different difficulty levels by synthetically adding random labels.
References in corpus (23)
- Scikit-learn: Machine Learning in Python
- On the Convergence of Adam and Beyond
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Three Factors Influencing Minima in SGD
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
- Recent Advances in Deep Learning: An Overview
- On the Convergence Rate of Training Recurrent Neural Networks
- Gradient Descent Happens in a Tiny Subspace
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
- A Walk with SGD
- Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
- Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
- Estimating Structured Vector Autoregressive Model
- Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks
- Escaping Saddle Points with Adaptive Gradient Methods
- A Scale Invariant Flatness Measure for Deep Network Minima
- Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis
- Positively Scale-Invariant Flatness of ReLU Neural Networks