On the Calibration of Multiclass Classification with Rejection

Abstract

We investigate the problem of multiclass classification with rejection, where a classifier can choose not to make a prediction to avoid critical misclassification. First, we consider an approach based on simultaneous training of a classifier and a rejector, which achieves the state-of-the-art performance in the binary case. We analyze this approach for the multiclass case and derive a general condition for calibration to the Bayes-optimal solution, which suggests that calibration is hard to achieve by general loss functions unlike the binary case. Next, we consider another traditional approach based on confidence scores, in which the existing work focuses on a specific class of losses. We propose rejection criteria for more general losses for this approach and guarantee calibration to the Bayes-optimal solution. Finally, we conduct experiments to validate the relevance of our theoretical findings.

arXiv commentNeurIPS2019 camera-ready, 31 pages

Cited by in corpus (6)

• Classification with Rejection Based on Cost-sensitive Classification
• ATRO: Adversarial Training with a Rejection Option
• On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspective
• Selective Classification via One-Sided Prediction
• When in Doubt, Summon the Titans: Efficient Inference with Large Models
• A Symmetric Loss Perspective of Reliable Machine Learning