'In-Between' Uncertainty in Bayesian Neural Networks

Abstract

We describe a limitation in the expressiveness of the predictive uncertainty estimate given by mean-field variational inference (MFVI), a popular approximate inference method for Bayesian neural networks. In particular, MFVI fails to give calibrated uncertainty estimates in between separated regions of observations. This can lead to catastrophically overconfident predictions when testing on out-of-distribution data. Avoiding such overconfidence is critical for active learning, Bayesian optimisation and out-of-distribution robustness. We instead find that a classical technique, the linearised Laplace approximation, can handle 'in-between' uncertainty much better for small network architectures.

Presented at the ICML 2019 Workshop on Uncertainty and Robustness in Deep Learning

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Deep Neural Networks as Gaussian Processes
• Variational Dropout and the Local Reparameterization Trick
• Scalable Bayesian Optimization Using Deep Neural Networks
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Overpruning in Variational Bayesian Neural Networks
• On the Importance of Strong Baselines in Bayesian Deep Learning

Cited by in corpus (23)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data
• Generalized Out-of-Distribution Detection: A Survey
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Laplace Redux -- Effortless Bayesian Deep Learning
• Subspace Inference for Bayesian Deep Learning
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Benchmarking the Neural Linear Model for Regression
• Estimating Model Uncertainty of Neural Networks in Sparse Information Form
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
• Bayesian Deep Learning via Subnetwork Inference
• Improving predictions of Bayesian neural nets via local linearization
• Bayesian Neural Networks: Essentials
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• Learnable Uncertainty under Laplace Approximations
• Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• A Novel Regression Loss for Non-Parametric Uncertainty Optimization
• Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data
• Understanding Uncertainty in Bayesian Deep Learning
• TyXe: Pyro-based Bayesian neural nets for Pytorch
• Estimating the Uncertainty of Neural Network Forecasts for Influenza Prevalence Using Web Search Activity
• Towards On-Chip Bayesian Neuromorphic Learning