Practical Evaluation of Out-of-Distribution Detection Methods for Image Classification
arXiv:2101.02447
Abstract
We reconsider the evaluation of OOD detection methods for image recognition. Although many studies have been conducted so far to build better OOD detection methods, most of them follow Hendrycks and Gimpel's work for the method of experimental evaluation. While the unified evaluation method is necessary for a fair comparison, there is a question of if its choice of tasks and datasets reflect real-world applications and if the evaluation results can generalize to other OOD detection application scenarios. In this paper, we experimentally evaluate the performance of representative OOD detection methods for three scenarios, i.e., irrelevant input detection, novel class detection, and domain shift detection, on various datasets and classification tasks. The results show that differences in scenarios and datasets alter the relative performance among the methods. Our results can also be used as a guide for practitioners for the selection of OOD detection methods.
References in corpus (7)
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
- Using Pre-Training Can Improve Model Robustness and Uncertainty
- A Public Image Database for Benchmark of Plant Seedling Classification Algorithms
- Unsupervised Domain Adaptation in Semantic Segmentation: a Review
- Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine Similarity