CoCa: Contrastive Captioners are Image-Text Foundation Models

Abstract

Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.

Preprint

Cited by in corpus (24)

• Reproducible scaling laws for contrastive language-image learning
• Language-aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classification
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• Physical Adversarial Attack meets Computer Vision: A Decade Survey
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Exploring scalable medical image encoders beyond text supervision
• Language Model Crossover: Variation through Few-Shot Prompting
• Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model
• CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model
• Multimodal Foundation Models for Material Property Prediction and Discovery
• Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval
• RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task
• Multimodal Neural Databases
• Adversarial Representation Learning for Robust Privacy Preservation in Audio
• Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
• Semi-supervised Multimodal Representation Learning through a Global Workspace
• MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
• ContextCLIP: Contextual Alignment of Image-Text pairs on CLIP visual representations
• Controlling the Output of a Generative Model by Latent Feature Vector Shifting
• GPU Memory Usage Optimization for Backward Propagation in Deep Network Training
• Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
• GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models