Bayesian Neural Networks with Soft Evidence

Abstract

Bayes's rule deals with hard evidence, that is, we can calculate the probability of event $A$ occuring given that event $B$ has occurred. Soft evidence, on the other hand, involves a degree of uncertainty about whether event $B$ has actually occurred or not. Jeffrey's rule of conditioning provides a way to update beliefs in the case of soft evidence. We provide a framework to learn a probability distribution on the weights of a neural network trained using soft evidence by way of two simple algorithms for approximating Jeffrey conditionalization. We propose an experimental protocol for benchmarking these algorithms on empirical datasets and find that Jeffrey based methods are competitive or better in terms of accuracy yet show improvements in calibration metrics upwards of 20% in some cases, even when the data contains mislabeled points.

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Deep Learning is Robust to Massive Label Noise
• Logistic Regression, Neural Networks and Dempster-Shafer Theory: a New Perspective