Does the Data Induce Capacity Control in Deep Learning?
arXiv:2110.14163
Abstract
We show that the input correlation matrix of typical classification datasets has an eigenspectrum where, after a sharp initial drop, a large number of small eigenvalues are distributed uniformly over an exponentially large range. This structure is mirrored in a network trained on this data: we show that the Hessian and the Fisher Information Matrix (FIM) have eigenvalues that are spread uniformly over exponentially large ranges. We call such eigenspectra "sloppy" because sets of weights corresponding to small eigenvalues can be changed by large magnitudes without affecting the loss. Networks trained on atypical datasets with non-sloppy inputs do not share these traits and deep networks trained on such datasets generalize poorly. Inspired by this, we study the hypothesis that sloppiness of inputs aids generalization in deep networks. We show that if the Hessian is sloppy, we can compute non-vacuous PAC-Bayes generalization bounds analytically. By exploiting our empirical observation that training predominantly takes place in the non-sloppy subspace of the FIM, we develop data-distribution dependent PAC-Bayes priors that lead to accurate generalization bounds using numerical optimization.
References in corpus (17)
- Universally Sloppy Parameter Sensitivities in Systems Biology
- Benign Overfitting in Linear Regression
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- The statistical mechanics of complex signaling networks : nerve growth factor signaling
- On Lazy Training in Differentiable Programming
- The geometry of nonlinear least squares with applications to sloppy models and optimization
- The sloppy model universality class and the Vandermonde matrix
- Sharp Minima Can Generalize For Deep Nets
- Gradient Descent Happens in a Tiny Subspace
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
- Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
- Practical Gauss-Newton Optimisation for Deep Learning
- The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
- Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
- Emergent properties of the local geometry of neural loss landscapes
- Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks