A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift

Abstract

The uncertainty estimation is critical in real-world decision making applications, especially when distributional shift between the training and test data are prevalent. Many calibration methods in the literature have been proposed to improve the predictive uncertainty of DNNs which are generally not well-calibrated. However, none of them is specifically designed to work properly under domain shift condition. In this paper, we propose Unsupervised Temperature Scaling (UTS) as a robust calibration method to domain shift. It exploits unlabeled test samples instead of the training one to adjust the uncertainty prediction of deep models towards the test distribution. UTS utilizes a novel loss function, weighted NLL, which allows unsupervised calibration. We evaluate UTS on a wide range of model-datasets to show the possibility of calibration without labels and demonstrate the robustness of UTS compared to other methods (e.g., TS, MC-dropout, SVI, ensembles) in shifted domains.

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• End to End Learning for Self-Driving Cars
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Variational Dropout Sparsifies Deep Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Bayesian Dark Knowledge
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Introspective Classification with Convolutional Nets