Scaling Out-of-Distribution Detection for Real-World Settings

Abstract

Detecting out-of-distribution examples is important for safety-critical machine learning applications such as detecting novel biological phenomena and self-driving cars. However, existing research mainly focuses on simple small-scale settings. To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and thousands of classes. To make future work in real-world settings possible, we create new benchmarks for three large-scale settings. To test ImageNet multiclass anomaly detectors, we introduce the Species dataset containing over 700,000 images and over a thousand anomalous species. We leverage ImageNet-21K to evaluate PASCAL VOC and COCO multilabel anomaly detectors. Third, we introduce a new benchmark for anomaly segmentation by introducing a segmentation benchmark with road anomalies. We conduct extensive experiments in these more realistic settings for out-of-distribution detection and find that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale multi-class, multi-label, and segmentation tasks, establishing a simple new baseline for future work.

arXiv commentICML 2022; The Species dataset and code are available at https://github.com/hendrycks/anomaly-seg

References in corpus (11)

• Deep Residual Learning for Image Recognition
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Deep Anomaly Detection with Outlier Exposure
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning
• Discriminative out-of-distribution detection for semantic segmentation
• A Meta-Analysis of the Anomaly Detection Problem
• Robust Semantic Segmentation with Ladder-DenseNet Models

Cited by in corpus (21)

• Natural Adversarial Examples
• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
• How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
• Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• Unified Classification and Rejection: A One-versus-All Framework
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• Impact, Attention, Influence: Early Assessment of Autonomous Driving Datasets
• AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving
• Are CLIP features all you need for Universal Synthetic Image Origin Attribution?
• Deep Metric Learning for Open World Semantic Segmentation
• Types of Out-of-Distribution Texts and How to Detect Them
• Robust Anomaly Detection for Particle Physics Using Multi-Background Representation Learning
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Enclosing Prototypical Variational Autoencoder for Explainable Out-of-Distribution Detection
• Familiarity-Based Open-Set Recognition Under Adversarial Attacks
• Toward a Realistic Benchmark for Out-of-Distribution Detection
• On the detection of Out-Of-Distribution samples in Multiple Instance Learning
• Open-set 3D Object Detection
• Consensus Synergizes with Memory: A Simple Approach for Anomaly Segmentation in Urban Scenes