Improving Classifier Confidence using Lossy Label-Invariant Transformations
arXiv:2011.04182
Abstract
Providing reliable model uncertainty estimates is imperative to enabling robust decision making by autonomous agents and humans alike. While recently there have been significant advances in confidence calibration for trained models, examples with poor calibration persist in most calibrated models. Consequently, multiple techniques have been proposed that leverage label-invariant transformations of the input (i.e., an input manifold) to improve worst-case confidence calibration. However, manifold-based confidence calibration techniques generally do not scale and/or require expensive retraining when applied to models with large input spaces (e.g., ImageNet). In this paper, we present the recursive lossy label-invariant calibration (ReCal) technique that leverages label-invariant transformations of the input that induce a loss of discriminatory information to recursively group (and calibrate) inputs - without requiring model retraining. We show that ReCal outperforms other calibration methods on multiple datasets, especially, on large-scale datasets such as ImageNet.
References in corpus (14)
- Playing Atari with Deep Reinforcement Learning
- Wide Residual Networks
- On Calibration of Modern Neural Networks
- Manifold Mixup: Better Representations by Interpolating Hidden States
- Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
- Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
- Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
- Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning
- Noisy Natural Gradient as Variational Inference
- Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks
- Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
- Classification Confidence Estimation with Test-Time Data-Augmentation
- On-manifold Adversarial Data Augmentation Improves Uncertainty Calibration
- Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network