Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization

Abstract

This paper deals with uncertainty quantification and out-of-distribution detection in deep learning using Bayesian and ensemble methods. It proposes a practical solution to the lack of prediction diversity observed recently for standard approaches when used out-of-distribution (Ovadia et al., 2019; Liu et al., 2021). Considering that this issue is mainly related to a lack of weight diversity, we claim that standard methods sample in "over-restricted" regions of the weight space due to the use of "over-regularization" processes, such as weight decay and zero-mean centered Gaussian priors. We propose to solve the problem by adopting the maximum entropy principle for the weight distribution, with the underlying idea to maximize the weight diversity. Under this paradigm, the epistemic uncertainty is described by the weight distribution of maximal entropy that produces neural networks "consistent" with the training observations. Considering stochastic neural networks, a practical optimization is derived to build such a distribution, defined as a trade-off between the average empirical risk and the weight distribution entropy. We develop a novel weight parameterization for the stochastic model, based on the singular value decomposition of the neural network's hidden representations, which enables a large increase of the weight entropy for a small empirical risk penalization. We provide both theoretical and numerical results to assess the efficiency of the approach. In particular, the proposed algorithm appears in the top three best methods in all configurations of an extensive out-of-distribution detection benchmark including more than thirty competitors.

References in corpus (48)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Variational Inference with Normalizing Flows
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Evidential Deep Learning to Quantify Classification Uncertainty
• Energy-based Out-of-distribution Detection
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Adversarial Reciprocal Points Learning for Open Set Recognition
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Towards Out-Of-Distribution Generalization: A Survey
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Scaling Out-of-Distribution Detection for Real-World Settings
• Practical Deep Learning with Bayesian Principles
• Functional Variational Bayesian Neural Networks
• OpenOOD: Benchmarking Generalized Out-of-Distribution Detection
• VOS: Learning What You Don't Know by Virtual Outlier Synthesis
• Mitigating Neural Network Overconfidence with Logit Normalization
• The Case for Bayesian Deep Learning
• Implicit Weight Uncertainty in Neural Networks
• Maximum Entropy Generators for Energy-Based Models
• Out-of-Distribution Detection with Deep Nearest Neighbors
• Global optimization of Lipschitz functions
• A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness
• The Functional Neural Process
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Optimal uncertainty quantification for legacy data observations of Lipschitz functions
• The Peril of Popular Deep Learning Uncertainty Estimation Methods
• Discrepancy-Based Active Learning for Domain Adaptation
• Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition
• Diversity regularization in deep ensembles
• Agree to Disagree: Diversity through Disagreement for Better Transferability
• Improving robustness and calibration in ensembles with diversity regularization
• Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification
• Robust uncertainty estimates with out-of-distribution pseudo-inputs training
• Semi-supervised novelty detection using ensembles with regularized disagreement
• Function-Space Regularization in Neural Networks: A Probabilistic Perspective