Information Aware Max-Norm Dirichlet Networks for Predictive Uncertainty Estimation

Abstract

Precise estimation of uncertainty in predictions for AI systems is a critical factor in ensuring trust and safety. Deep neural networks trained with a conventional method are prone to over-confident predictions. In contrast to Bayesian neural networks that learn approximate distributions on weights to infer prediction confidence, we propose a novel method, Information Aware Dirichlet networks, that learn an explicit Dirichlet prior distribution on predictive distributions by minimizing a bound on the expected max norm of the prediction error and penalizing information associated with incorrect outcomes. Properties of the new cost function are derived to indicate how improved uncertainty estimation is achieved. Experiments using real datasets show that our technique outperforms, by a large margin, state-of-the-art neural networks for estimating within-distribution and out-of-distribution uncertainty, and detecting adversarial examples.

arXiv commentTo appear in Neural Networks. https://doi.org/10.1016/j.neunet.2020.12.011

References in corpus (19)

• Sequence to Sequence Learning with Neural Networks
• TensorFlow: A system for large-scale machine learning
• Explaining and Harnessing Adversarial Examples
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• On Calibration of Modern Neural Networks
• Rényi Divergence and Kullback-Leibler Divergence
• Weight Uncertainty in Neural Networks
• Deep Learning for Identifying Metastatic Breast Cancer
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Evidential Deep Learning to Quantify Classification Uncertainty
• Bayesian Active Learning for Classification and Preference Learning
• Stochastic Gradient Hamiltonian Monte Carlo
• Variational Dropout and the Local Reparameterization Trick
• Variational Dropout Sparsifies Deep Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness

Cited by in corpus (4)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Extensible Machine Learning for Encrypted Network Traffic Application Labeling via Uncertainty Quantification
• Deep Evidential Regression
• Improving Explainable Object-induced Model through Uncertainty for Automated Vehicles