PAC-Bayes with Backprop
arXiv:1908.07380
Abstract
We explore the family of methods "PAC-Bayes with Backprop" (PBB) to train probabilistic neural networks by minimizing PAC-Bayes bounds. We present two training objectives, one derived from a previously known PAC-Bayes bound, and a second one derived from a novel PAC-Bayes bound. Both training objectives are evaluated on MNIST and on various UCI data sets. Our experiments show two striking observations: we obtain competitive test set error estimates (~1.4% on MNIST) and at the same time we compute non-vacuous bounds with much tighter values (~2.3% on MNIST) than previous results. These observations suggest that neural nets trained by PBB may lead to self-bounding learning, where the available data can be used to simultaneously learn a predictor and certify its risk, with no need to follow a data-splitting protocol.
References in corpus (7)
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Are All Layers Created Equal?
- A Note on the PAC Bayesian Theorem
- Pathwise Derivatives Beyond the Reparameterization Trick
- A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descent
- PAC-Bayesian Margin Bounds for Convolutional Neural Networks
Cited by in corpus (9)
- User-friendly introduction to PAC-Bayes bounds
- Tighter risk certificates for neural networks
- Stochastic Neural Network with Kronecker Flow
- PAC-Bayes Analysis Beyond the Usual Bounds
- Probably Approximately Correct Vision-Based Planning using Motion Primitives
- Deep kernel processes
- How Tight Can PAC-Bayes be in the Small Data Regime?
- Learning Provably Robust Motion Planners Using Funnel Libraries
- Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability