On the Expressiveness of Approximate Inference in Bayesian Neural Networks

Abstract

While Bayesian neural networks (BNNs) hold the promise of being flexible, well-calibrated statistical models, inference often requires approximations whose consequences are poorly understood. We study the quality of common variational methods in approximating the Bayesian predictive distribution. For single-hidden layer ReLU BNNs, we prove a fundamental limitation in function-space of two of the most commonly used distributions defined in weight-space: mean-field Gaussian and Monte Carlo dropout. We find there are simple cases where neither method can have substantially increased uncertainty in between well-separated regions of low uncertainty. We provide strong empirical evidence that exact inference does not have this pathology, hence it is due to the approximation and not the model. In contrast, for deep networks, we prove a universality result showing that there exist approximate posteriors in the above classes which provide flexible uncertainty estimates. However, we find empirically that pathologies of a similar form as in the single-hidden layer case can persist when performing variational inference in deeper networks. Our results motivate careful consideration of the implications of approximate inference methods in BNNs.

arXiv commentNeurIPS 2020 version

References in corpus (22)

• Auto-Encoding Variational Bayes
• Weight Uncertainty in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Deep Neural Networks as Gaussian Processes
• Stochastic Gradient Hamiltonian Monte Carlo
• Variational Dropout and the Local Reparameterization Trick
• Deep Ensembles: A Loss Landscape Perspective
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Practical Deep Learning with Bayesian Principles
• Functional Variational Bayesian Neural Networks
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Deep Convolutional Networks as shallow Gaussian Processes
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• Rényi Divergence Variational Inference
• Improving and Understanding Variational Continual Learning
• 'In-Between' Uncertainty in Bayesian Neural Networks
• On the Importance of Strong Baselines in Bayesian Deep Learning
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• Exact posterior distributions of wide Bayesian neural networks
• Scalable Training of Inference Networks for Gaussian-Process Models

Cited by in corpus (13)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• What Are Bayesian Neural Network Posteriors Really Like?
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Bayesian Neural Network Priors Revisited
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Benchmarking the Neural Linear Model for Regression
• Adaptive variational Bayes: Optimality, computation and applications
• On the role of Model Uncertainties in Bayesian Optimization
• The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction
• Quantifying Model Uncertainty in Inverse Problems via Bayesian Deep Gradient Descent