Information-Theoretic Local Minima Characterization and Regularization
arXiv:1911.08192
Abstract
Recent advances in deep learning theory have evoked the study of generalizability across different local minima of deep neural networks (DNNs). While current work focused on either discovering properties of good local minima or developing regularization techniques to induce good local minima, no approach exists that can tackle both problems. We achieve these two goals successfully in a unified manner. Specifically, based on the observed Fisher information we propose a metric both strongly indicative of generalizability of local minima and effectively applied as a practical regularizer. We provide theoretical analysis including a generalization bound and empirically demonstrate the success of our approach in both capturing and improving the generalizability of DNNs. Experiments are performed on CIFAR-10, CIFAR-100 and ImageNet for various network architectures.
arXiv commentICML 2020
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- The Loss Surfaces of Multilayer Networks
- Improving Generalization Performance by Switching from Adam to SGD
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Robust Large Margin Deep Neural Networks
- Averaging Weights Leads to Wider Optima and Better Generalization
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Spectrally-normalized margin bounds for neural networks
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Sharp Minima Can Generalize For Deep Nets
- Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
- Predicting the Generalization Gap in Deep Networks with Margin Distributions
- Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Skip Connections Eliminate Singularities
- Norm-Based Capacity Control in Neural Networks
- Optimization Landscape and Expressivity of Deep CNNs