Predictive Uncertainty Estimation via Prior Networks

Abstract

Estimating how uncertain an AI system is in its predictions is important to improve the safety of such systems. Uncertainty in predictive can result from uncertainty in model parameters, irreducible data uncertainty and uncertainty due to distributional mismatch between the test and training data distributions. Different actions might be taken depending on the source of the uncertainty so it is important to be able to distinguish between them. Recently, baseline tasks and metrics have been defined and several practical methods to estimate uncertainty developed. These methods, however, attempt to model uncertainty due to distributional mismatch either implicitly through model uncertainty or as data uncertainty. This work proposes a new framework for modeling predictive uncertainty called Prior Networks (PNs) which explicitly models distributional uncertainty. PNs do this by parameterizing a prior distribution over predictive distributions. This work focuses on uncertainty for classification and evaluates PNs on the tasks of identifying out-of-distribution (OOD) samples and detecting misclassification on the MNIST dataset, where they are found to outperform previous methods. Experiments on synthetic and MNIST and CIFAR-10 data show that unlike previous non-Bayesian methods PNs are able to distinguish between data and distributional uncertainty.

References in corpus (7)

• Efficient Estimation of Word Representations in Vector Space
• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks

Cited by in corpus (50)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Generalized Out-of-Distribution Detection: A Survey
• A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• AI Security for Geoscience and Remote Sensing: Challenges and Future Trends
• Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization
• $\mathbf{\gamma }$-Net: Superresolving SAR Tomographic Inversion via Deep Learning
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Reliability and Interpretability in Science and Deep Learning
• The Unreasonable Effectiveness of Deep Evidential Regression
• Entropic Out-of-Distribution Detection: Seamless Detection of Unknown Examples
• Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Variance Networks: When Expectation Does Not Meet Your Expectations
• JuryGCN: Quantifying Jackknife Uncertainty on Graph Convolutional Networks
• Quantifying Classification Uncertainty using Regularized Evidential Neural Networks
• DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inference
• SoK: Machine Learning Governance
• Predictive Uncertainty Quantification with Compound Density Networks
• Robust Node Classification on Graphs: Jointly from Bayesian Label Transition and Topology-based Label Propagation
• Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression
• Efficient Neural Architecture Search via Proximal Iterations
• Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
• Accurate Passive Radar via an Uncertainty-Aware Fusion of Wi-Fi Sensing Data
• One Versus all for deep Neural Network Incertitude (OVNNI) quantification
• Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over Simplex
• Controlling Risk of Web Question Answering
• Interpretable Uncertainty Quantification in AI for HEP
• Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• Confidence-Guided Learning Process for Continuous Classification of Time Series
• Modeling Token-level Uncertainty to Learn Unknown Concepts in SLU via Calibrated Dirichlet Prior RNN
• Uncertainty-Aware Deep Classifiers using Generative Models
• Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection
• Prior Networks for Detection of Adversarial Attacks
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
• Understanding the Under-Coverage Bias in Uncertainty Estimation
• On Out-of-distribution Detection with Energy-based Models
• RBUE: A ReLU-Based Uncertainty Estimation Method of Deep Neural Networks
• Uncertainty Propagation in Node Classification
• Out-of-distribution detection in satellite image classification
• Reliability Quantification of Deep Reinforcement Learning-based Control
• In the Search for Optimal Multi-view Learning Models for Crop Classification with Global Remote Sensing Data
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Structured DropConnect for Uncertainty Inference in Image Classification
• Uncertainty based Class Activation Maps for Visual Question Answering
• Toward a Realistic Benchmark for Out-of-Distribution Detection