Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections
arXiv:1905.13195
Abstract
Modeling uncertainty in deep neural networks, despite recent important advances, is still an open problem. Bayesian neural networks are a powerful solution, where the prior over network weights is a design choice, often a normal distribution or other distribution encouraging sparsity. However, this prior is agnostic to the generative process of the input data, which might lead to unwarranted generalization for out-of-distribution tested data. We suggest the presence of a confounder for the relation between the input data and the discriminative function given the target label. We propose an approach for modeling this confounder by sharing neural connectivity patterns between the generative and discriminative networks. This approach leads to a new deep architecture, where networks are sampled from the posterior of local causal structures, and coupled into a compact hierarchy. We demonstrate that sampling networks from this hierarchy, proportionally to their posterior, is efficient and enables estimating various types of uncertainties. Empirical evaluations of our method demonstrate significant improvement compared to state-of-the-art calibration and out-of-distribution detection methods.
References in corpus (10)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Auto-Encoding Variational Bayes
- On Calibration of Modern Neural Networks
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Weight Uncertainty in Neural Networks
- Learning Confidence for Out-of-Distribution Detection in Neural Networks
- Averaging Weights Leads to Wider Optima and Better Generalization
- A Simple Baseline for Bayesian Uncertainty in Deep Learning
- Constructing Deep Neural Networks by Bayesian Network Structure Learning
- Bayesian Structure Learning by Recursive Bootstrap