Re-Scoring Using Image-Language Similarity for Few-Shot Object Detection

Abstract

Few-shot object detection, which focuses on detecting novel objects with few labels, is an emerging challenge in the community. Recent studies show that adapting a pre-trained model or modified loss function can improve performance. In this paper, we explore leveraging the power of Contrastive Language-Image Pre-training (CLIP) and hard negative classification loss in low data setting. Specifically, we propose Re-scoring using Image-language Similarity for Few-shot object detection (RISF) which extends Faster R-CNN by introducing Calibration Module using CLIP (CM-CLIP) and Background Negative Re-scale Loss (BNRL). The former adapts CLIP, which performs zero-shot classification, to re-score the classification scores of a detector using image-class similarities, the latter is modified classification loss considering the punishment for fake backgrounds as well as confusing categories on a generalized few-shot object detection dataset. Extensive experiments on MS-COCO and PASCAL VOC show that the proposed RISF substantially outperforms the state-of-the-art approaches. The code will be available.

arXiv comment19 pages, 11 figures

References in corpus (8)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• BEiT: BERT Pre-Training of Image Transformers
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
• MM-FSOD: Meta and metric integrated few-shot object detection