Identifying Generalization Properties in Neural Networks
arXiv:1809.07402
Abstract
While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order "smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of the model, as well as an algorithm that optimizes the perturbed model accordingly.
arXiv comment23 pages
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- The Natural Language Decathlon: Multitask Learning as Question Answering
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
Cited by in corpus (18)
- Are All Layers Created Equal?
- Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
- Understanding Generalization through Visualizations
- A Scale Invariant Flatness Measure for Deep Network Minima
- Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
- Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis
- SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
- Optimization Variance: Exploring Generalization Properties of DNNs
- Can VAEs Generate Novel Examples?
- Deforming the Loss Surface to Affect the Behaviour of the Optimizer
- A Reparameterization-Invariant Flatness Measure for Deep Neural Networks
- Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation
- The role of invariance in spectral complexity-based generalization bounds
- Deforming the Loss Surface
- Minimum sharpness: Scale-invariant parameter-robustness of neural networks
- LRTuner: A Learning Rate Tuner for Deep Neural Networks
- Relative Flatness and Generalization
- BN-invariant sharpness regularizes the training model to better generalization