Calibrating sufficiently

Abstract

When probabilistic classifiers are trained and calibrated, the so-called grouping loss component of the calibration loss can easily be overlooked. Grouping loss refers to the gap between observable information and information actually exploited in the calibration exercise. We investigate the relation between grouping loss and the concept of sufficiency, identifying comonotonicity as a useful criterion for sufficiency. We revisit the probing reduction approach of Langford & Zadrozny (2005) and find that it produces an estimator of probabilistic classifiers that reduces grouping loss. Finally, we discuss Brier curves as tools to support training and 'sufficient' calibration of probabilistic classifiers.

arXiv comment27 pages, 2 figures, appendix

References in corpus (5)

• Reliability, Sufficiency, and the Decomposition of Proper Scores
• The role of the information set for forecasting - with applications to risk management
• Predicting accurate probabilities with a ranking loss
• Mitigating Bias in Calibration Error Estimation
• A plug-in approach to maximising precision at the top and recall at the top

Cited by in corpus (1)

• Minimising quantifier variance under prior probability shift