Confidence Calibration with Bounded Error Using Transformations

Abstract

As machine learning techniques become widely adopted in new domains, especially in safety-critical systems such as autonomous vehicles, it is crucial to provide accurate output uncertainty estimation. As a result, many approaches have been proposed to calibrate neural networks to accurately estimate the likelihood of misclassification. However, while these methods achieve low calibration error, there is space for further improvement, especially in large-dimensional settings such as ImageNet. In this paper, we introduce a calibration algorithm, named Hoki, that works by applying random transformations to the neural network logits. We provide a sufficient condition for perfect calibration based on the number of label prediction changes observed after applying the transformations. We perform experiments on multiple datasets and show that the proposed approach generally outperforms state-of-the-art calibration algorithms across multiple datasets and models, especially on the challenging ImageNet dataset. Finally, Hoki is scalable as well, as it requires comparable execution time to that of temperature scaling.

References in corpus (9)

• YOLOv4: Optimal Speed and Accuracy of Object Detection
• On Calibration of Modern Neural Networks
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
• Measuring Calibration in Deep Learning
• Verified Uncertainty Calibration
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
• Classification Confidence Estimation with Test-Time Data-Augmentation
• PAC Confidence Predictions for Deep Neural Network Classifiers
• Improving Classifier Confidence using Lossy Label-Invariant Transformations