Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation
arXiv:1909.03009
Abstract
Explaining how overparametrized neural networks simultaneously achieve low risk and zero empirical risk on benchmark datasets is an open problem. PAC-Bayes bounds optimized using variational inference (VI) have been recently proposed as a promising direction in obtaining non-vacuous bounds. We show empirically that this approach gives negligible gains when modeling the posterior as a Gaussian with diagonal covariance--known as the mean-field approximation. We investigate common explanations, such as the failure of VI due to problems in optimization or choosing a suboptimal prior. Our results suggest that investigating richer posteriors is the most promising direction forward.
References in corpus (19)
- Auto-Encoding Variational Bayes
- Variational Inference with Normalizing Flows
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Variational Dropout and the Local Reparameterization Trick
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
- Sharp Minima Can Generalize For Deep Nets
- Uniform convergence may be unable to explain generalization in deep learning
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Identifying Generalization Properties in Neural Networks
- The Description Length of Deep Learning Models
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
- SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
- A Scale Invariant Flatness Measure for Deep Network Minima
- Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis
- Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
- Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
- Stochastic Neural Network with Kronecker Flow
- Information-Theoretic Local Minima Characterization and Regularization