Calibrated Top-1 Uncertainty estimates for classification by score based models

Abstract

While the accuracy of modern deep learning models has significantly improved in recent years, the ability of these models to generate uncertainty estimates has not progressed to the same degree. Uncertainty methods are designed to provide an estimate of class probabilities when predicting class assignment. While there are a number of proposed methods for estimating uncertainty, they all suffer from a lack of calibration: predicted probabilities can be off from empirical ones by a few percent or more. By restricting the scope of our predictions to only the probability of Top-1 error, we can decrease the calibration error of existing methods to less than one percent. As a result, the scores of the methods also improve significantly over benchmarks.

arXiv comment12 pages, 5 figures, 6 tables (major revision, new benchmark allows us to show model calibration is better)

References in corpus (5)

• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Variational Dropout and the Local Reparameterization Trick
• Identifying Mislabeled Data using the Area Under the Margin Ranking