Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

Abstract

Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. In this paper, we introduce a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. Unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. In order to improve learning from noisy web data, we propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. We provide a theoretical analysis of ALBEF from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. ALBEF achieves state-of-the-art performance on multiple downstream vision-language tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained on orders of magnitude larger datasets. On VQA and NLVR${}^2$, ALBEF achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-art, while enjoying faster inference speed. Code and pre-trained models are available at https://github.com/salesforce/ALBEF/.

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• Learning Transferable Visual Models From Natural Language Supervision
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• A large annotated corpus for learning natural language inference
• Born Again Neural Networks
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Unifying Vision-and-Language Tasks via Text Generation
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Bilinear Attention Networks
• Modeling Context in Referring Expressions
• e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations
• Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

Cited by in corpus (36)

• Florence: A New Foundation Model for Computer Vision
• VLP: A Survey on Vision-Language Pre-training
• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Search
• IMF: Interactive Multimodal Fusion Model for Link Prediction
• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model
• Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
• Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval
• Multimodal Foundation Models for Material Property Prediction and Discovery
• Mobile Foundation Model as Firmware
• When Search Meets Recommendation: Learning Disentangled Search Representation for Recommendation
• Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval
• Transforming Visual Scene Graphs to Image Captions
• 3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting
• MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks
• Cross-Modal Retrieval for Motion and Text via DropTriple Loss
• Multi-queue Momentum Contrast for Microvideo-Product Retrieval
• Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task
• RTQ: Rethinking Video-language Understanding Based on Image-text Model
• Automatic Medical Report Generation: Methods and Applications
• OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network
• Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
• IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training
• STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
• Image-Text Pre-Training for Logo Recognition
• MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
• ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation
• Multi-Modality Transformer for E-Commerce: Inferring User Purchase Intention to Bridge the Query-Product Gap
• printf: Preference Modeling Based on User Reviews with Item Images and Textual Information via Graph Learning
• sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging
• PØDA: Prompt-driven Zero-shot Domain Adaptation
• Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval