Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection

Abstract

Inspired by the success of vision-language methods (VLMs) in zero-shot classification, recent works attempt to extend this line of work into object detection by leveraging the localization ability of pre-trained VLMs and generating pseudo labels for unseen classes in a self-training manner. However, since the current VLMs are usually pre-trained with aligning sentence embedding with global image embedding, the direct use of them lacks fine-grained alignment for object instances, which is the core of detection. In this paper, we propose a simple but effective fine-grained Visual-Text Prompt-driven self-training paradigm for Open-Vocabulary Detection (VTP-OVD) that introduces a fine-grained visual-text prompt adapting stage to enhance the current self-training paradigm with a more powerful fine-grained alignment. During the adapting stage, we enable VLM to obtain fine-grained alignment by using learnable text prompts to resolve an auxiliary dense pixel-wise prediction task. Furthermore, we propose a visual prompt module to provide the prior task information (i.e., the categories need to be predicted) for the vision branch to better adapt the pre-trained VLM to the downstream tasks. Experiments show that our method achieves the state-of-the-art performance for open-vocabulary object detection, e.g., 31.5% mAP on unseen classes of COCO.

References in corpus (12)

• Learning to Prompt for Vision-Language Models
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Zero-Shot Text-to-Image Generation
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
• ActionCLIP: A New Paradigm for Video Action Recognition
• Semantics-Guided Contrastive Network for Zero-Shot Object detection
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
• Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection
• M6: A Chinese Multimodal Pretrainer
• ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation

Cited by in corpus (2)

• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• Contextual Emotion Recognition using Large Vision Language Models