Soft Calibration Objectives for Neural Networks
arXiv:2108.00106
Abstract
Optimal decision making requires that classifiers produce uncertainty estimates consistent with their empirical accuracy. However, deep neural networks are often under- or over-confident in their predictions. Consequently, methods have been developed to improve the calibration of their predictive uncertainty both during training and post-hoc. In this work, we propose differentiable losses to improve calibration based on a soft (continuous) version of the binning operation underlying popular calibration-error estimators. When incorporated into training, these soft calibration losses achieve state-of-the-art single-model ECE across multiple datasets with less than 1% decrease in accuracy. For instance, we observe an 82% reduction in ECE (70% relative to the post-hoc rescaled ECE) in exchange for a 0.7% relative decrease in accuracy relative to the cross entropy baseline on CIFAR-100. When incorporated post-training, the soft-binning-based calibration error objective improves upon temperature scaling, a popular recalibration method. Overall, experiments across losses and datasets demonstrate that using calibration-sensitive procedures yield better uncertainty estimates under dataset shift than the standard practice of using a cross entropy loss and post-hoc recalibration methods.
arXiv comment17 pages total, 10 page main paper, 5 page appendix, 10 figures total, 8 figures in main paper, 2 figures in appendix
References in corpus (13)
- Deep Residual Learning for Image Recognition
- End to End Learning for Self-Driving Cars
- On Calibration of Modern Neural Networks
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- When Does Label Smoothing Help?
- On Fairness and Calibration
- Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
- Calibrating Deep Neural Networks using Focal Loss
- Improving model calibration with accuracy versus uncertainty optimization
- Mitigating Bias in Calibration Error Estimation
- Estimating Uncertainty Online Against an Adversary
- Calibration of Neural Networks using Splines
- Local Interpretability of Calibrated Prediction Models: A Case of Type 2 Diabetes Mellitus Screening Test