Tighter risk certificates for neural networks

Abstract

This paper presents an empirical study regarding training probabilistic neural networks using training objectives derived from PAC-Bayes bounds. In the context of probabilistic neural networks, the output of training is a probability distribution over network weights. We present two training objectives, used here for the first time in connection with training neural networks. These two training objectives are derived from tight PAC-Bayes bounds. We also re-implement a previously used training objective based on a classical PAC-Bayes bound, to compare the properties of the predictors learned using the different training objectives. We compute risk certificates for the learnt predictors, based on part of the data used to learn the predictors. We further experiment with different types of priors on the weights (both data-free and data-dependent priors) and neural network architectures. Our experiments on MNIST and CIFAR-10 show that our training methods produce competitive test set errors and non-vacuous risk bounds with much tighter values than previous results in the literature, showing promise not only to guide the learning algorithm through bounding the risk but also for model selection. These observations suggest that the methods studied here might be good candidates for self-certified learning, in the sense of using the whole data set for learning a predictor and certifying its risk on any unseen data (from the same distribution as the training data) potentially without the need for holding out test data.

arXiv commentNew version includes: i) experiment showing the potential of the risk certificate for neural architecture search (Fig. 2); ii) experiments spanning uncertainty quantification and analysis of prior/posterior (Section 7.8); iii) an outline of the strengths of probabilistic neural networks trained by PBB (Section 7.9) and iv) a strengthened discussion on the connection to Bayesian learning

References in corpus (21)

• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Practical Deep Learning with Bayesian Principles
• Generalization in Adaptive Data Analysis and Holdout Reuse
• A Note on the PAC Bayesian Theorem
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Risk Bounds for the Majority Vote: From a PAC-Bayesian Analysis to a Learning Algorithm
• On the properties of variational approximations of Gibbs posteriors
• A PAC-Bayesian Tutorial with A Dropout Bound
• Pathwise Derivatives Beyond the Reparameterization Trick
• A Primer on PAC-Bayesian Learning
• A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descent
• The Generalized Reparameterization Gradient
• A Strongly Quasiconvex PAC-Bayesian Bound
• On the role of data in PAC-Bayes bounds
• PAC-Bayes with Backprop
• PAC-Bayes Mini-tutorial: A Continuous Union Bound
• PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons
• Unifying Variational Inference and PAC-Bayes for Supervised Learning that Scales

Cited by in corpus (15)

• User-friendly introduction to PAC-Bayes bounds
• Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees
• Learning PAC-Bayes Priors for Probabilistic Neural Networks
• PAC-Bayes Bounds for Meta-learning with Data-Dependent Prior
• PAC-Bayes Analysis Beyond the Usual Bounds
• PAC-Bayes Bounds for Bandit Problems: A Survey and Experimental Comparison
• Learning via Wasserstein-Based High Probability Generalisation Bounds
• Formal limitations of sample-wise information-theoretic generalization bounds
• Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation
• Wide stochastic networks: Gaussian limit and PAC-Bayesian training
• How Tight Can PAC-Bayes be in the Small Data Regime?
• Generalisation under gradient descent via deterministic PAC-Bayes
• Comparing Comparators in Generalization Bounds
• Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability
• A note on regularised NTK dynamics with an application to PAC-Bayesian training