Exploring the Limits of Out-of-Distribution Detection

Abstract

Near out-of-distribution detection (OOD) is a major challenge for deep neural networks. We demonstrate that large-scale pre-trained transformers can significantly improve the state-of-the-art (SOTA) on a range of near OOD tasks across different data modalities. For instance, on CIFAR-100 vs CIFAR-10 OOD detection, we improve the AUROC from 85% (current SOTA) to more than 96% using Vision Transformers pre-trained on ImageNet-21k. On a challenging genomics OOD detection benchmark, we improve the AUROC from 66% to 77% using transformers and unsupervised pre-training. To further improve performance, we explore the few-shot outlier exposure setting where a few examples from outlier classes may be available; we show that pre-trained transformers are particularly well-suited for outlier exposure, and that the AUROC of OOD detection on CIFAR-100 vs CIFAR-10 can be improved to 98.7% with just 1 image per OOD class, and 99.46% with 10 images per OOD class. For multi-modal image-text pre-trained transformers such as CLIP, we explore a new way of using just the names of outlier classes as a sole source of information without any accompanying images, and show that this outperforms previous SOTA on standard vision OOD benchmark tasks.

S.F. and J.R. contributed equally

References in corpus (6)

• Learning Transferable Visual Models From Natural Language Supervision
• Language Models are Few-Shot Learners
• On Calibration of Modern Neural Networks
• Deep Anomaly Detection with Outlier Exposure
• Revisiting the Calibration of Modern Neural Networks
• Contrastive Training for Improved Out-of-Distribution Detection

Cited by in corpus (10)

• Generalized Out-of-Distribution Detection: A Survey
• A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Statistical process monitoring of artificial neural networks
• Understanding the Natural Language of DNA using Encoder-Decoder Foundation Models with Byte-level Precision
• For a semiotic AI: Bridging computer vision and visual semiotics for computational observation of large scale facial image archives
• Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
• Sparse MoEs meet Efficient Ensembles
• Toward a Realistic Benchmark for Out-of-Distribution Detection