What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?

Abstract

Understanding classifier decision under novel environments is central to the community, and a common practice is evaluating it on labeled test sets. However, in real-world testing, image annotations are difficult and expensive to obtain, especially when the test environment is changing. A natural question then arises: given a trained classifier, can we evaluate its accuracy on varying unlabeled test sets? In this work, we train semantic classification and rotation prediction in a multi-task way. On a series of datasets, we report an interesting finding, i.e., the semantic classification accuracy exhibits a strong linear relationship with the accuracy of the rotation prediction task (Pearson's Correlation r > 0.88). This finding allows us to utilize linear regression to estimate classifier performance from the accuracy of rotation prediction which can be obtained on the test set through the freely generated rotation labels.

arXiv commentICML 2021 camera ready

References in corpus (9)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• VisDA: The Visual Domain Adaptation Challenge
• Unsupervised Domain Adaptation through Self-Supervision
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Predicting Neural Network Accuracy from Weights
• Classifying the classifier: dissecting the weight space of neural networks