How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
arXiv:2306.06048 · doi:10.1007/s11263-023-01895-7
Abstract
Recent large vision-language models such as CLIP have shown remarkable out-of-distribution (OOD) detection and generalization performance. However, their zero-shot in-distribution (ID) accuracy is often limited for downstream datasets. Recent CLIP-based fine-tuning methods such as prompt learning have demonstrated significant improvements in ID classification and OOD generalization where OOD labels are available. Nonetheless, it remains unclear whether the model is reliable to semantic shifts without OOD labels. In this paper, we aim to bridge the gap and present a comprehensive study to understand how fine-tuning impact OOD detection for few-shot downstream tasks. By framing OOD detection as multi-modal concept matching, we establish a connection between fine-tuning methods and various OOD scores. Our results suggest that a proper choice of OOD scores is essential for CLIP-based fine-tuning. In particular, the maximum concept matching (MCM) score provides a promising solution consistently. We also show that prompt learning demonstrates the state-of-the-art OOD detection performance over the zero-shot counterpart.
Accepted to IJCV 2023
References in corpus (10)
- Learning Transferable Visual Models From Natural Language Supervision
- Learning to Prompt for Vision-Language Models
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- FILIP: Fine-grained Interactive Language-Image Pre-Training
- Scaling Out-of-Distribution Detection for Real-World Settings
- Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
- Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
- Delving into Out-of-Distribution Detection with Vision-Language Representations