How Much Can CLIP Benefit Vision-and-Language Tasks?

Abstract

Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.

arXiv comment14 pages

References in corpus (18)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Learning Transferable Visual Models From Natural Language Supervision
• Microsoft COCO Captions: Data Collection and Evaluation Server
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• VQA: Visual Question Answering
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Discriminability objective for training descriptive captions
• A Better Variant of Self-Critical Sequence Training
• Language and Visual Entity Relationship Graph for Agent Navigation
• Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

Cited by in corpus (8)

• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• MERLOT: Multimodal Neural Script Knowledge Models
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation
• EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling
• IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training
• Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval