A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks
arXiv:2112.00337 · doi:10.1016/j.eswa.2023.120461
Abstract
Deep neural networks have achieved outstanding performance over various tasks, but they have a critical issue: over-confident predictions even for completely unknown samples. Many studies have been proposed to successfully filter out these unknown samples, but they only considered narrow and specific tasks, referred to as misclassification detection, open-set recognition, or out-of-distribution detection. In this work, we argue that these tasks should be treated as fundamentally an identical problem because an ideal model should possess detection capability for all those tasks. Therefore, we introduce the unknown detection task, an integration of previous individual tasks, for a rigorous examination of the detection capability of deep neural networks on a wide spectrum of unknown samples. To this end, unified benchmark datasets on different scales were constructed and the unknown detection capabilities of existing popular methods were subject to comparison. We found that Deep Ensemble consistently outperforms the other approaches in detecting unknowns; however, all methods are only successful for a specific type of unknown. The reproducible code and benchmark datasets are available at https://github.com/daintlab/unknown-detection-benchmarks .
arXiv commentPublished in ESWA (https://www.sciencedirect.com/science/article/pii/S0957417423009636)
References in corpus (24)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- A Simple Framework for Contrastive Learning of Visual Representations
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
- Evidential Deep Learning to Quantify Classification Uncertainty
- Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation
- Learning Confidence for Out-of-Distribution Detection in Neural Networks
- CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
- Selective Classification for Deep Neural Networks
- WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
- Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
- Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
- Do Deep Generative Models Know What They Don't Know?
- Practical Solutions for Machine Learning Safety in Autonomous Vehicles
- Out-of-distribution Detection in Classifiers via Generation
- Confidence Calibration for Convolutional Neural Networks Using Structured Dropout
- Are Out-of-Distribution Detection Methods Effective on Large-Scale Datasets?
- Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
- One-vs-Rest Network-based Deep Probability Model for Open Set Recognition
- No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets