Quality of Uncertainty Quantification for Bayesian Neural Network Inference

Abstract

Bayesian Neural Networks (BNNs) place priors over the parameters in a neural network. Inference in BNNs, however, is difficult; all inference methods for BNNs are approximate. In this work, we empirically compare the quality of predictive uncertainty estimates for 10 common inference methods on both regression and classification tasks. Our experiments demonstrate that commonly used metrics (e.g. test log-likelihood) can be misleading. Our experiments also indicate that inference innovations designed to capture structure in the posterior do not necessarily produce high quality posterior approximations.

Accepted to ICML UDL 2019

References in corpus (6)

• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Single-Model Uncertainties for Deep Learning
• Variational Boosting: Iteratively Refining Posterior Approximations

Cited by in corpus (32)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data
• Multi-fidelity Bayesian Neural Networks: Algorithms and Applications
• What Are Bayesian Neural Network Posteriors Really Like?
• Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• Subspace Inference for Bayesian Deep Learning
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Physics-integrated machine learning: embedding a neural network in the Navier-Stokes equations. Part I
• Physics-integrated machine learning: embedding a neural network in the Navier-Stokes equations. Part II
• Benchmarking the Neural Linear Model for Regression
• Deep Bayesian Gaussian Processes for Uncertainty Estimation in Electronic Health Records
• Dual Accuracy-Quality-Driven Neural Network for Prediction Interval Generation
• Prediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles
• Deep Ensembles from a Bayesian Perspective
• The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks
• Bayesian Autoencoders: Analysing and Fixing the Bernoulli likelihood for Out-of-Distribution Detection
• Uncertainty Quantification and Software Risk Analysis for Digital Twins in the Nearly Autonomous Management and Control Systems: A Review
• Sample-efficient reinforcement learning using deep Gaussian processes
• Horseshoe Regularization for Machine Learning in Complex and Deep Models
• Uncertainty Characteristics Curves: A Systematic Assessment of Prediction Intervals
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• Avoiding Tampering Incentives in Deep RL via Decoupled Approval
• Understanding Uncertainty in Bayesian Deep Learning
• CRAUM-Net: Contextual Recursive Attention with Uncertainty Modeling for Salient Object Detection
• Coalitional Bayesian Autoencoders -- Towards explainable unsupervised deep learning
• StelNet: Hierarchical Neural Network for Automatic Inference in Stellar Characterization
• Generative Particle Variational Inference via Estimation of Functional Gradients
• Decision Theoretic Bootstrapping
• Automated Detection of Congenital Heart Disease in Fetal Ultrasound Screening
• Estimating the Uncertainty of Neural Network Forecasts for Influenza Prevalence Using Web Search Activity