ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Abstract

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.

arXiv comment11 pages, 5 figures

References in corpus (8)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Cross-lingual Language Model Pretraining
• Modulating early visual processing by language
• VideoBERT: A Joint Model for Video and Language Representation Learning
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• MAttNet: Modular Attention Network for Referring Expression Comprehension
• Colorization as a Proxy Task for Visual Understanding
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering

Cited by in corpus (114)

• Pre-trained Models for Natural Language Processing: A Survey
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• VLP: A Survey on Vision-Language Pre-training
• Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion
• Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• Adversarial Text-to-Image Synthesis: A Review
• VLT: Vision-Language Transformer and Query Generation for Referring Segmentation
• Self-Supervised Learning for Videos: A Survey
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• Local-Global Context Aware Transformer for Language-Guided Video Segmentation
• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• A Deep Multi-Level Attentive network for Multimodal Sentiment Analysis
• Calibration of Transformer-based Models for Identifying Stress and Depression in Social Media
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Disentangling Hate in Online Memes
• Representation Learning for Natural Language Processing
• Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering
• Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering
• From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
• VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
• Context Disentangling and Prototype Inheriting for Robust Visual Grounding
• Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
• Supervised Contrastive Learning for Multimodal Unreliable News Detection in COVID-19 Pandemic
• Parallel Attention Network with Sequence Matching for Video Grounding
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• Gender and Racial Bias in Visual Question Answering Datasets
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamese
• Universal Multimodal Representation for Language Understanding
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• On Distinctive Image Captioning via Comparing and Reweighting
• Interpreting Black Box Models via Hypothesis Testing
• VD-BERT: A Unified Vision and Dialog Transformer with BERT
• A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and method
• RepsNet: Combining Vision with Language for Automated Medical Reports
• Discrete and continuous representations and processing in deep learning: Looking forward
• Visual Relationship Detection with Visual-Linguistic Knowledge from Multimodal Representations
• Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering
• ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design
• A Dataset of Alt Texts from HCI Publications: Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in Scientific Papers
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• Representation learning for neural population activity with Neural Data Transformers
• Self-Training Vision Language BERTs with a Unified Conditional Model
• Recommending Themes for Ad Creative Design via Visual-Linguistic Representations
• Masked Vision-Language Transformer in Fashion
• Image Retrieval from Contextual Descriptions
• MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks
• Mobile App Crowdsourced Test Report Consistency Detection via Deep Image-and-Text Fusion Understanding
• Exploiting Natural Language for Efficient Risk-Aware Multi-robot SaR Planning
• MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• Pareto Invariant Representation Learning for Multimedia Recommendation
• Tile Classification Based Viewport Prediction with Multi-modal Fusion Transformer
• Linguistically-aware Attention for Reducing the Semantic-Gap in Vision-Language Tasks
• VD-PCR: Improving Visual Dialog with Pronoun Coreference Resolution
• Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detection
• Effective End-to-End Vision Language Pretraining with Semantic Visual Loss
• Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation
• Sentiment Analysis of Fashion Related Posts in Social Media
• Subverting Fair Image Search with Generative Adversarial Perturbations
• OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network
• Comprehensive Multi-Modal Interactions for Referring Image Segmentation
• MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigation
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• Collaborative Three-Stream Transformers for Video Captioning
• Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration
• IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training
• Interactive Video Corpus Moment Retrieval using Reinforcement Learning
• Transformers for End-to-End InfoSec Tasks: A Feasibility Study
• Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
• Generalizable Neuro-symbolic Systems for Commonsense Question Answering
• Automatic Generation of Product-Image Sequence in E-commerce
• Probing the Role of Positional Information in Vision-Language Models
• Visual Grounding of Whole Radiology Reports for 3D CT Images
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
• ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer
• VLSlice: Interactive Vision-and-Language Slice Discovery
• Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation
• PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
• Unified Multi-modal Diagnostic Framework with Reconstruction Pre-training and Heterogeneity-combat Tuning
• ConditionNET: Learning Preconditions and Effects for Execution Monitoring
• Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
• An Entropy Clustering Approach for Assessing Visual Question Difficulty
• Rephrasing visual questions by specifying the entropy of the answer distribution
• ROSGPT_Vision: Commanding Robots Using Only Language Models' Prompts
• Context-Aware Group Captioning via Self-Attention and Contrastive Features
• AI as a Tool for Fair Journalism: Case Studies from Malta
• Supervised Visual Attention for Simultaneous Multimodal Machine Translation
• A Multimodal Single-Branch Embedding Network for Recommendation in Cold-Start and Missing Modality Scenarios
• Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings
• A Multi-level Alignment Training Scheme for Video-and-Language Grounding
• Local Slot Attention for Vision-and-Language Navigation
• Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition
• Enhancing Cross-Modal Contextual Congruence for Crowdfunding Success using Knowledge-infused Learning
• Multi-Modality Transformer for E-Commerce: Inferring User Purchase Intention to Bridge the Query-Product Gap
• Open-Ended Multi-Modal Relational Reasoning for Video Question Answering
• Curriculum Learning for Compositional Visual Reasoning
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
• printf: Preference Modeling Based on User Reviews with Item Images and Textual Information via Graph Learning
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
• When Did It Happen? Duration-informed Temporal Localization of Narrated Actions in Vlogs
• Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
• Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models