Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networks

Abstract

We make three related contributions motivated by the challenge of training stochastic neural networks, particularly in a PAC-Bayesian setting: (1) we show how averaging over an ensemble of stochastic neural networks enables a new class of \emph{partially-aggregated} estimators; (2) we show that these lead to provably lower-variance gradient estimates for non-differentiable signed-output networks; (3) we reformulate a PAC-Bayesian bound for these networks to derive a directly optimisable, differentiable objective and a generalisation guarantee, without using a surrogate loss or loosening the bound. This bound is twice as tight as that of Letarte et al. (2019) on a similar network type. We show empirically that these innovations make training easier and lead to competitive guarantees.

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning

Cited by in corpus (4)

• Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation
• Learning via Wasserstein-Based High Probability Generalisation Bounds
• Comparing Comparators in Generalization Bounds
• Generalisation under gradient descent via deterministic PAC-Bayes