Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks

Abstract

Recently, Deep Neural Networks (DNNs) have been achieving impressive results on wide range of tasks. However, they suffer from being well-calibrated. In decision-making applications, such as autonomous driving or medical diagnosing, the confidence of deep networks plays an important role to bring the trust and reliability to the system. To calibrate the deep networks' confidence, many probabilistic and measure-based approaches are proposed. Temperature Scaling (TS) is a state-of-the-art among measure-based calibration methods which has low time and memory complexity as well as effectiveness. In this paper, we study TS and show it does not work properly when the validation set that TS uses for calibration has small size or contains noisy-labeled samples. TS also cannot calibrate highly accurate networks as well as non-highly accurate ones. Accordingly, we propose Attended Temperature Scaling (ATS) which preserves the advantages of TS while improves calibration in aforementioned challenging situations. We provide theoretical justifications for ATS and assess its effectiveness on wide range of deep models and datasets. We also compare the calibration results of TS and ATS on skin lesion detection application as a practical problem where well-calibrated system can play important role in making a decision.

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• End to End Learning for Self-Driving Cars
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Variational Dropout Sparsifies Deep Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Bayesian Dark Knowledge
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Introspective Classification with Convolutional Nets

Cited by in corpus (3)

• Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• How do you feel? Measuring User-Perceived Value for Rejecting Machine Decisions in Hate Speech Detection