Calibration tests in multi-class classification: A unifying framework

Abstract

In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class classification that generalize existing measures such as the expected calibration error, the maximum calibration error, and the maximum mean calibration error. We propose and evaluate empirically different consistent and unbiased estimators for a specific class of measures based on matrix-valued kernels. Importantly, these estimators can be interpreted as test statistics associated with well-defined bounds and approximations of the p-value under the null hypothesis that the model is calibrated, significantly improving the interpretability of calibration measures, which otherwise lack any meaningful unit or scale.

Corrected version that 1) fixes the ECE evaluation with bins of uniform size (does not affect our conclusions and discussions) and 2) contains additional experimental results in the supplementary material

Cited by in corpus (19)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Classifier Calibration: A survey on how to assess and improve predicted class probabilities
• Verified Uncertainty Calibration
• Calibrating Deep Neural Networks using Focal Loss
• Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification
• Assessing Generalization of SGD via Disagreement
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
• Should Ensemble Members Be Calibrated?
• Combining Ensembles and Data Augmentation can Harm your Calibration
• Distribution-free binary classification: prediction sets, confidence intervals and calibration
• Distribution-free calibration guarantees for histogram binning without sample splitting
• Calibration of Neural Networks using Splines
• Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration
• Top-label calibration and multiclass-to-binary reductions
• On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifiers
• Calibration tests beyond classification
• Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain
• Identifying and Exploiting Structures for Reliable Deep Learning
• On Deep Neural Network Calibration by Regularization and its Impact on Refinement