All You Need is a Good Functional Prior for Bayesian Deep Learning

Abstract

The Bayesian treatment of neural networks dictates that a prior distribution is specified over their weight and bias parameters. This poses a challenge because modern neural networks are characterized by a large number of parameters, and the choice of these priors has an uncontrolled effect on the induced functional prior, which is the distribution of the functions obtained by sampling the parameters from their prior distribution. We argue that this is a hugely limiting aspect of Bayesian deep learning, and this work tackles this limitation in a practical and effective way. Our proposal is to reason in terms of functional priors, which are easier to elicit, and to "tune" the priors of neural network parameters in a way that they reflect such functional priors. Gaussian processes offer a rigorous framework to define prior distributions over functions, and we propose a novel and robust framework to match their prior with the functional prior of neural networks based on the minimization of their Wasserstein distance. We provide vast experimental evidence that coupling these priors with scalable Markov chain Monte Carlo sampling offers systematically large performance improvements over alternative choices of priors and state-of-the-art approximate Bayesian deep learning approaches. We consider this work a considerable step in the direction of making the long-standing challenge of carrying out a fully Bayesian treatment of neural networks, including convolutional neural networks, a concrete possibility.

References in corpus (23)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Variational Inference with Normalizing Flows
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Stochastic Gradient Hamiltonian Monte Carlo
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Practical Deep Learning with Bayesian Principles
• Functional Variational Bayesian Neural Networks
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
• Avoiding pathologies in very deep networks
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• Pathwise Derivatives Beyond the Reparameterization Trick
• Noise Contrastive Priors for Functional Uncertainty
• The Deep Weight Prior
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Good Initializations of Variational Bayes for Deep Models
• Output-Constrained Bayesian Neural Networks
• The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks

Cited by in corpus (5)

• What Are Bayesian Neural Network Posteriors Really Like?
• Learning Functional Priors and Posteriors from Data and Physics
• Bayesian Neural Network Priors Revisited
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Revisiting the Effects of Stochasticity for Hamiltonian Samplers