Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness

Abstract

Bayesian neural networks (BNN) and deep ensembles are principled approaches to estimate the predictive uncertainty of a deep learning model. However their practicality in real-time, industrial-scale applications are limited due to their heavy memory and inference cost. This motivates us to study principled approaches to high-quality uncertainty estimation that require only a single deep neural network (DNN). By formalizing the uncertainty quantification as a minimax learning problem, we first identify input distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data in the input space, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs, by adding a weight normalization step during training and replacing the output layer with a Gaussian process. On a suite of vision and language understanding tasks and on modern architectures (Wide-ResNet and BERT), SNGP is competitive with deep ensembles in prediction, calibration and out-of-domain detection, and outperforms the other single-model approaches.

References in corpus (31)

• Spectral Normalization for Generative Adversarial Networks
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Evidential Deep Learning to Quantify Classification Uncertainty
• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• Scalable Bayesian Optimization Using Deep Neural Networks
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• Robust Large Margin Deep Neural Networks
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• Measuring Calibration in Deep Learning
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• Stochastic Variational Deep Kernel Learning
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
• Single-Model Uncertainties for Deep Learning
• Excessive Invariance Causes Adversarial Vulnerability
• i-RevNet: Deep Invertible Networks
• Finite Sample Bernstein -- von Mises Theorem for Semiparametric Problems
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection
• DOC: Deep Open Classification of Text Documents
• Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
• Representing smooth functions as compositions of near-identity functions with implications for deep network optimization
• Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables
• Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)
• Towards Open Intent Discovery for Conversational Text
• Prior Networks for Detection of Adversarial Attacks
• A deterministic and computable Bernstein-von Mises theorem

Cited by in corpus (29)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• An Uncertainty-aware Transfer Learning-based Framework for Covid-19 Diagnosis
• On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty
• Kepler: Robust Learning for Faster Parametric Query Optimization
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Feature Space Singularity for Out-of-Distribution Detection
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• A case for new neural network smoothness constraints
• Deep Deterministic Uncertainty for Semantic Segmentation
• The Hidden Uncertainty in a Neural Networks Activations
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
• Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
• Can convolutional ResNets approximately preserve input distances? A frequency analysis perspective
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• DeDUCE: Generating Counterfactual Explanations Efficiently
• BEDS-Bench: Behavior of EHR-models under Distributional Shift--A Benchmark
• RoMA: Robust Model Adaptation for Offline Model-based Optimization
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• Understanding Uncertainty in Bayesian Deep Learning
• Towards Robust Active Feature Acquisition
• Query-based Hard-Image Retrieval for Object Detection at Test Time
• Test Sample Accuracy Scales with Training Sample Density in Neural Networks
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty
• $\Delta$-UQ: Accurate Uncertainty Quantification via Anchor Marginalization
• FROB: Few-shot ROBust Model for Classification and Out-of-Distribution Detection
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• What classifiers know what they don't?