Exploring the Limits of Out-of-Distribution Detection

Abstract

Near out-of-distribution detection (OOD) is a major challenge for deep neural networks. We demonstrate that large-scale pre-trained transformers can significantly improve the state-of-the-art (SOTA) on a range of near OOD tasks across different data modalities. For instance, on CIFAR-100 vs CIFAR-10 OOD detection, we improve the AUROC from 85% (current SOTA) to more than 96% using Vision Transformers pre-trained on ImageNet-21k. On a challenging genomics OOD detection benchmark, we improve the AUROC from 66% to 77% using transformers and unsupervised pre-training. To further improve performance, we explore the few-shot outlier exposure setting where a few examples from outlier classes may be available; we show that pre-trained transformers are particularly well-suited for outlier exposure, and that the AUROC of OOD detection on CIFAR-100 vs CIFAR-10 can be improved to 98.7% with just 1 image per OOD class, and 99.46% with 10 images per OOD class. For multi-modal image-text pre-trained transformers such as CLIP, we explore a new way of using just the names of outlier classes as a sole source of information without any accompanying images, and show that this outperforms previous SOTA on standard vision OOD benchmark tasks.

arXiv commentS.F. and J.R. contributed equally

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• Learning Transferable Visual Models From Natural Language Supervision
• Language Models are Few-Shot Learners
• Shortcut Learning in Deep Neural Networks
• On Calibration of Modern Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Semi-supervised Sequence Learning
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Deep Anomaly Detection with Outlier Exposure
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
• Revisiting the Calibration of Modern Neural Networks
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
• Contrastive Training for Improved Out-of-Distribution Detection
• Towards Robust Vision Transformer

Cited by in corpus (12)

• Generalized Out-of-Distribution Detection: A Survey
• Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP
• A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection
• Dynamic Defense Against Byzantine Poisoning Attacks in Federated Learning
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Statistical process monitoring of artificial neural networks
• Understanding the Natural Language of DNA using Encoder-Decoder Foundation Models with Byte-level Precision
• For a semiotic AI: Bridging computer vision and visual semiotics for computational observation of large scale facial image archives
• Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
• Sparse MoEs meet Efficient Ensembles
• Toward a Realistic Benchmark for Out-of-Distribution Detection