Scaling Out-of-Distribution Detection for Real-World Settings

Abstract

Detecting out-of-distribution examples is important for safety-critical machine learning applications such as detecting novel biological phenomena and self-driving cars. However, existing research mainly focuses on simple small-scale settings. To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and thousands of classes. To make future work in real-world settings possible, we create new benchmarks for three large-scale settings. To test ImageNet multiclass anomaly detectors, we introduce the Species dataset containing over 700,000 images and over a thousand anomalous species. We leverage ImageNet-21K to evaluate PASCAL VOC and COCO multilabel anomaly detectors. Third, we introduce a new benchmark for anomaly segmentation by introducing a segmentation benchmark with road anomalies. We conduct extensive experiments in these more realistic settings for out-of-distribution detection and find that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale multi-class, multi-label, and segmentation tasks, establishing a simple new baseline for future work.

ICML 2022; The Species dataset and code are available at https://github.com/hendrycks/anomaly-seg

References in corpus (1)

• Deep Anomaly Detection with Outlier Exposure

Cited by in corpus (20)

• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
• How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
• Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• Unified Classification and Rejection: A One-versus-All Framework
• Impact, Attention, Influence: Early Assessment of Autonomous Driving Datasets
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving
• Are CLIP features all you need for Universal Synthetic Image Origin Attribution?
• Deep Metric Learning for Open World Semantic Segmentation
• Types of Out-of-Distribution Texts and How to Detect Them
• Robust Anomaly Detection for Particle Physics Using Multi-Background Representation Learning
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Consensus Synergizes with Memory: A Simple Approach for Anomaly Segmentation in Urban Scenes
• Familiarity-Based Open-Set Recognition Under Adversarial Attacks
• Open-set 3D Object Detection
• On the detection of Out-Of-Distribution samples in Multiple Instance Learning
• Toward a Realistic Benchmark for Out-of-Distribution Detection
• Enclosing Prototypical Variational Autoencoder for Explainable Out-of-Distribution Detection