Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
arXiv:2105.04290
Abstract
In many applications, it is desirable that a classifier not only makes accurate predictions, but also outputs calibrated posterior probabilities. However, many existing classifiers, especially deep neural network classifiers, tend to be uncalibrated. Post-hoc calibration is a technique to recalibrate a model by learning a calibration map. Existing approaches mostly focus on constructing calibration maps with low calibration errors, however, this quality is inadequate for a calibrator being useful. In this paper, we introduce two constraints that are worth consideration in designing a calibration map for post-hoc calibration. Then we present Meta-Cal, which is built from a base calibrator and a ranking model. Under some mild assumptions, two high-probability bounds are given with respect to these constraints. Empirical results on CIFAR-10, CIFAR-100 and ImageNet and a range of popular network architectures show our proposed method significantly outperforms the current state of the art for post-hoc multi-class classification calibration.
References in corpus (13)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Deep Residual Learning for Image Recognition
- Densely Connected Convolutional Networks
- Wide Residual Networks
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Predictive Uncertainty Estimation via Prior Networks
- Selective Classification for Deep Neural Networks
- A Simple Baseline for Bayesian Uncertainty in Deep Learning
- Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
- Neyman-Pearson (NP) classification algorithms and NP receiver operating characteristics (NP-ROC)
- Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning
- Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning