Contrastive Out-of-Distribution Detection for Pretrained Transformers

Abstract

Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time. Therefore, in practice, a reliable model should identify such instances, and then either reject them during inference or pass them over to models that handle another distribution. In this paper, we develop an unsupervised OOD detection method, in which only the in-distribution (ID) data are used in training. We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones. These OOD instances can then be accurately detected using the Mahalanobis distance in the model's penultimate layer. We experiment with comprehensive settings and achieve near-perfect OOD detection performance, outperforming baselines drastically. We further investigate the rationales behind the improvement, finding that more compact representations through margin-based contrastive learning bring the improvement. We release our code to the community for future research.

arXiv commentAccepted at EMNLP 2021. Code available at https://github.com/wzhouad/Contra-OOD

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Deep Visual Domain Adaptation: A Survey
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Energy-based Out-of-distribution Detection
• Deep Anomaly Detection with Outlier Exposure
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Classification-Based Anomaly Detection for General Data
• Reducing Network Agnostophobia
• Contrastive Training for Improved Out-of-Distribution Detection

Cited by in corpus (4)

• GOOD-D: On Unsupervised Graph Out-Of-Distribution Detection
• VendorLink: An NLP approach for Identifying & Linking Vendor Migrants & Potential Aliases on Darknet Markets
• Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Data
• Types of Out-of-Distribution Texts and How to Detect Them