Predictive Uncertainty Quantification with Compound Density Networks

Abstract

Despite the huge success of deep neural networks (NNs), finding good mechanisms for quantifying their prediction uncertainty is still an open problem. Bayesian neural networks are one of the most popular approaches to uncertainty quantification. On the other hand, it was recently shown that ensembles of NNs, which belong to the class of mixture models, can be used to quantify prediction uncertainty. In this paper, we build upon these two approaches. First, we increase the mixture model's flexibility by replacing the fixed mixing weights by an adaptive, input-dependent distribution (specifying the probability of each component) represented by NNs, and by considering uncountably many mixture components. The resulting class of models can be seen as the continuous counterpart to mixture density networks and is therefore referred to as compound density networks (CDNs). We employ both maximum likelihood and variational Bayesian inference to train CDNs, and empirically show that they yield better uncertainty estimates on out-of-distribution data and are more robust to adversarial examples than the previous approaches.

arXiv commentBayesian deep learning workshop, NeuRIPS 2019

References in corpus (17)

• Auto-Encoding Variational Bayes
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Natural Language Processing (almost) from Scratch
• The information bottleneck method
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Predictive Uncertainty Estimation via Prior Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Implicit Weight Uncertainty in Neural Networks
• Uncertainty in the Variational Information Bottleneck
• Bayesian Hypernetworks
• Stochastic Maximum Likelihood Optimization via Hypernetworks

Cited by in corpus (5)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Principled Weight Initialization for Hypernetworks
• Probabilistic Deep Learning with Probabilistic Neural Networks and Deep Probabilistic Models
• Evaluating Scalable Uncertainty Estimation Methods for DNN-Based Molecular Property Prediction
• Investigating maximum likelihood based training of infinite mixtures for uncertainty quantification