VisualBERT: A Simple and Performant Baseline for Vision and Language

Abstract

We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks. VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention. We further propose two visually-grounded language model objectives for pre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2, and Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly simpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.

Work in Progress

References in corpus (9)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Multimodal Transformer with Multi-View Visual Representation for Image Captioning
• MUREL: Multimodal Relational Reasoning for Visual Question Answering

Cited by in corpus (251)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Pre-trained Models for Natural Language Processing: A Survey
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• MetaFormer Baselines for Vision
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• VLP: A Survey on Vision-Language Pre-training
• Attention Bottlenecks for Multimodal Fusion
• Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion
• Learning Video Representations using Contrastive Bidirectional Transformer
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• Adversarial Text-to-Image Synthesis: A Review
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• UNITER: UNiversal Image-TExt Representation Learning
• Towards Natural Language Interfaces for Data Visualization: A Survey
• Supervised Multimodal Bitransformers for Classifying Images and Text
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• Language-aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classification
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Information Retrieval: Recent Advances and Beyond
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Biomedical Question Answering: A Survey of Approaches and Challenges
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
• Referring Segmentation in Images and Videos with Cross-Modal Self-Attention Network
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Disentangling Hate in Online Memes
• Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
• Efficient Self-supervised Vision Transformers for Representation Learning
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering
• Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks
• VinVL: Revisiting Visual Representations in Vision-Language Models
• Emotion Detection for Misinformation: A Review
• MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
• Multiscale Vision Transformers
• InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
• VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
• MERLOT: Multimodal Neural Script Knowledge Models
• Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge
• Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• M6: A Chinese Multimodal Pretrainer
• Modality-Agnostic Attention Fusion for visual search with text feedback
• A Multimodal Framework for the Detection of Hateful Memes
• Active Contrastive Learning of Audio-Visual Video Representations
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• Unified Pre-training for Program Understanding and Generation
• Detecting Hate Speech in Multi-modal Memes
• A Recurrent Vision-and-Language BERT for Navigation
• Component Segmentation of Engineering Drawings Using Graph Convolutional Networks
• Support-set bottlenecks for video-text representation learning
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Gender and Racial Bias in Visual Question Answering Datasets
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• A Multimodal Memes Classification: A Survey and Open Research Issues
• OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamese
• Robustness Verification for Transformers
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
• A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and method
• VD-BERT: A Unified Vision and Dialog Transformer with BERT
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• MiniVLM: A Smaller and Faster Vision-Language Model
• UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
• Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
• Non-autoregressive Transformer-based End-to-end ASR using BERT
• Visual Relationship Detection with Visual-Linguistic Knowledge from Multimodal Representations
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• MusCaps: Generating Captions for Music Audio
• ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design
• A Dataset of Alt Texts from HCI Publications: Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in Scientific Papers
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• Self-Training Vision Language BERTs with a Unified Conditional Model
• Learning the Best Pooling Strategy for Visual Semantic Embedding
• Neural Language Generation: Formulation, Methods, and Evaluation
• Human-Adversarial Visual Question Answering
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• From Data Deluge to Data Curation: A Filtering-WoRA Paradigm for Efficient Text-based Person Search
• Knowledge Perceived Multi-modal Pretraining in E-commerce
• Recommending Themes for Ad Creative Design via Visual-Linguistic Representations
• OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
• MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering
• In Defense of Grid Features for Visual Question Answering
• Contrastive Visual-Linguistic Pretraining
• Open-Vocabulary Object Detection Using Captions
• MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation
• Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
• Language Grounding with 3D Objects
• Multimodal Contrastive Training for Visual Representation Learning
• A Context-aware Attention and Graph Neural Network-based Multimodal Framework for Misogyny Detection
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• Scaling Up Vision-Language Pre-training for Image Captioning
• Pre-Trained Models: Past, Present and Future
• Human-centric Spatio-Temporal Video Grounding With Visual Transformers
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• Hateful Memes Detection via Complementary Visual and Linguistic Networks
• A Survey of Multimodal Sarcasm Detection
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detection
• A Survey on Multimodal Disinformation Detection
• Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
• TRIE: End-to-End Text Reading and Information Extraction for Document Understanding
• Multi-modal Extreme Classification
• Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation
• Towards a Wireless Physical-Layer Foundation Model: Challenges and Strategies
• Deep Multimodal Neural Architecture Search
• Learning from Lexical Perturbations for Consistent Visual Question Answering
• IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
• Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
• Multimodal Fusion Refiner Networks
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• Cross-media Structured Common Space for Multimedia Event Extraction
• Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• What Vision-Language Models `See' when they See Scenes
• Attention on Abstract Visual Reasoning
• Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• NLVR2 Visual Bias Analysis
• Attention over learned object embeddings enables complex visual reasoning
• IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training
• HateProof: Are Hateful Meme Detection Systems really Robust?
• All-in-One Image-Grounded Conversational Agents
• Metadata-based Multi-Task Bandits with Bayesian Hierarchical Models
• LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding
• Multimodal Conditionality for Natural Language Generation
• A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
• LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
• Towards General Purpose Vision Systems
• KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for Visual Commonsense Reasoning
• Automatic Generation of Product-Image Sequence in E-commerce
• VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
• Visual Question Answering on 360° Images
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
• CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations
• Causal Attention for Vision-Language Tasks
• Multimodal Learning for Hateful Memes Detection
• A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension
• GlyphCRM: Bidirectional Encoder Representation for Chinese Character with its Glyph
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• Inferring spatial relations from textual descriptions of images
• CapWAP: Captioning with a Purpose
• ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration
• IITP at WAT 2021: System description for English-Hindi Multimodal Translation Task
• A Picture May Be Worth a Hundred Words for Visual Question Answering
• Probing Image-Language Transformers for Verb Understanding
• Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
• Multimodal End-to-End Sparse Model for Emotion Recognition
• Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Images
• Predicting Opioid Use Disorder from Longitudinal Healthcare Data using Multi-stream Transformer
• LAMP: Label Augmented Multimodal Pretraining
• Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
• Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
• Learning to Represent Image and Text with Denotation Graph
• Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation
• Context-Aware Group Captioning via Self-Attention and Contrastive Features
• VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words
• Grounding Spatial Relations in Text-Only Language Models
• AI as a Tool for Fair Journalism: Case Studies from Malta
• MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation
• Check It Again: Progressive Visual Question Answering via Visual Entailment
• MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification
• Enhance Multimodal Model Performance with Data Augmentation: Facebook Hateful Meme Challenge Solution
• Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• DVCFlow: Modeling Information Flow Towards Human-like Video Captioning
• Curriculum Learning for Compositional Visual Reasoning
• MATK: The Meme Analytical Tool Kit
• Latent Alignment of Procedural Concepts in Multimodal Recipes
• A Multi-level Alignment Training Scheme for Video-and-Language Grounding
• 15 Keypoints Is All You Need
• Chop Chop BERT: Visual Question Answering by Chopping VisualBERT's Heads
• FinEmbedDiff: A Cost-Effective Approach of Classifying Financial Documents with Vector Sampling using Multi-modal Embedding Models
• HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
• printf: Preference Modeling Based on User Reviews with Item Images and Textual Information via Graph Learning
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
• Modeling Coreference Relations in Visual Dialog
• Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventions
• A Transformer-based Cross-modal Fusion Model with Adversarial Training for VQA Challenge 2021
• Separating Skills and Concepts for Novel Visual Question Answering
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Answer-checking in Context: A Multi-modal FullyAttention Network for Visual Question Answering
• Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining
• Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
• Understanding of Emotion Perception from Art
• Hierarchical Similarity Learning for Language-based Product Image Retrieval
• Can images help recognize entities? A study of the role of images for Multimodal NER
• Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training
• Fine-Grained Chemical Entity Typing with Multimodal Knowledge Representation
• Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
• Data Efficient Masked Language Modeling for Vision and Language
• Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering
• Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers
• Multi-stage Pre-training over Simplified Multimodal Pre-training Models
• Panoptic Narrative Grounding
• Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning
• Cross-Modal Retrieval Augmentation for Multi-Modal Classification
• Caption Enriched Samples for Improving Hateful Memes Detection
• COVR: A test-bed for Visually Grounded Compositional Generalization with real images
• MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets
• Detecting Harmful Memes and Their Targets
• ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns' Semantic Properties and their Prototypicality
• Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models
• Utilizing Every Image Object for Semi-supervised Phrase Grounding
• Transductive Zero-Shot Learning using Cross-Modal CycleGAN
• Improving Calibration in Deep Metric Learning With Cross-Example Softmax
• DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework
• AOMD: An Analogy-aware Approach to Offensive Meme Detection on Social Media
• Cross-Modality Relevance for Reasoning on Language and Vision
• MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding
• Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models
• Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
• Interpretable Multimodal Out-of-context Detection with Soft Logic Regularization
• Detecting Propaganda Techniques in Memes
• Analysis on Image Set Visual Question Answering
• Beyond Language: Learning Commonsense from Images for Reasoning
• Adversarial Evaluation of Multimodal Models under Realistic Gray Box Assumption