Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks

Abstract

Accurate estimation of predictive uncertainty in modern neural networks is critical to achieve well calibrated predictions and detect out-of-distribution (OOD) inputs. The most promising approaches have been predominantly focused on improving model uncertainty (e.g. deep ensembles and Bayesian neural networks) and post-processing techniques for OOD detection (e.g. ODIN and Mahalanobis distance). However, there has been relatively little investigation into how the parametrization of the probabilities in discriminative classifiers affects the uncertainty estimates, and the dominant method, softmax cross-entropy, results in misleadingly high confidences on OOD data and under covariate shift. We investigate alternative ways of formulating probabilities using (1) a one-vs-all formulation to capture the notion of "none of the above", and (2) a distance-based logit representation to encode uncertainty as a function of distance to the training manifold. We show that one-vs-all formulations can improve calibration on image classification tasks, while matching the predictive performance of softmax without incurring any additional training or test-time complexity.

References in corpus (11)

• End to End Learning for Self-Driving Cars
• Wide Residual Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Weight Uncertainty in Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Large Batch Training of Convolutional Networks
• Composite Binary Losses
• Image Classification Using SVMs: One-against-One Vs One-against-All
• Deep-RBF Networks Revisited: Robust Classification with Rejection
• Small and Practical BERT Models for Sequence Labeling
• One Versus all for deep Neural Network Incertitude (OVNNI) quantification

Cited by in corpus (6)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• OVANet: One-vs-All Network for Universal Domain Adaptation
• Unified Classification and Rejection: A One-versus-All Framework
• Distributional Gaussian Process Layers for Outlier Detection in Image Segmentation
• Evaluation of Out-of-Distribution Detection Performance of Self-Supervised Learning in a Controllable Environment