Confidence from Invariance to Image Transformations

Abstract

We develop a technique for automatically detecting the classification errors of a pre-trained visual classifier. Our method is agnostic to the form of the classifier, requiring access only to classifier responses to a set of inputs. We train a parametric binary classifier (error/correct) on a representation derived from a set of classifier responses generated from multiple copies of the same input, each subject to a different natural image transformation. Thus, we establish a measure of confidence in classifier's decision by analyzing the invariance of its decision under various transformations. In experiments with multiple data sets (STL-10,CIFAR-100,ImageNet) and classifiers, we demonstrate new state of the art for the error detection task. In addition, we apply our technique to novelty detection scenarios, where we also demonstrate state of the art results.

References in corpus (4)

• Distilling the Knowledge in a Neural Network
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Countering Adversarial Images using Input Transformations
• Safer Classification by Synthesis

Cited by in corpus (5)

• Anomalous Example Detection in Deep Learning: A Survey
• Controlling Risk of Web Question Answering
• Classification Confidence Estimation with Test-Time Data-Augmentation
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts