Classifier Calibration: A survey on how to assess and improve predicted class probabilities

Abstract

This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.

arXiv commentMachine Learning (2023)

References in corpus (18)

• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
• Fast calibrated additive quantile regression
• Evaluating model calibration in classification
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification
• Loss-Calibrated Approximate Inference in Bayesian Neural Networks
• Calibration tests in multi-class classification: A unifying framework
• Calibrated Reliable Regression using Maximum Mean Discrepancy
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• Calibrating Deep Convolutional Gaussian Processes
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
• Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation
• Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
• Ensemble-based Uncertainty Quantification: Bayesian versus Credal Inference

Cited by in corpus (7)

• AutoSourceID-Classifier. Star-Galaxy Classification using a Convolutional Neural Network with Spatial Information
• Selection of powerful radio galaxies with machine learning
• Robust quantum dots charge autotuning using neural network uncertainty
• Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results
• Calibration through the Lens of Interpretability
• h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective
• Bayesian classification of astronomical spectra with class uncertainties