Understanding Uncertainty in Bayesian Deep Learning

Abstract

Neural Linear Models (NLM) are deep Bayesian models that produce predictive uncertainty by learning features from the data and then performing Bayesian linear regression over these features. Despite their popularity, few works have focused on formally evaluating the predictive uncertainties of these models. Furthermore, existing works point out the difficulties of encoding domain knowledge in models like NLMs, making them unsuitable for applications where interpretability is required. In this work, we show that traditional training procedures for NLMs can drastically underestimate uncertainty in data-scarce regions. We identify the underlying reasons for this behavior and propose a novel training method that can both capture useful predictive uncertainties as well as allow for incorporation of domain knowledge.

arXiv comment97 pages, 32 figures, Master of Engineering Thesis

References in corpus (13)

• Scikit-learn: Machine Learning in Python
• Weight Uncertainty in Neural Networks
• Scalable Bayesian Optimization Using Deep Neural Networks
• Functional Variational Bayesian Neural Networks
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• Thoughts on Massively Scalable Gaussian Processes
• Black Box Variational Inference
• Training independent subnetworks for robust prediction
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Benchmarking the Neural Linear Model for Regression
• Learning Qualitatively Diverse and Interpretable Rules for Classification
• Adaptive Bayesian Linear Regression for Automated Machine Learning