VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Abstract

We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{https://github.com/jackroos/VL-BERT}.

Accepted by ICLR 2020

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Cross-lingual Language Model Pretraining
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Learning Video Representations using Contrastive Bidirectional Transformer
• VideoBERT: A Joint Model for Video and Language Representation Learning
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Rethinking ImageNet Pre-training

Cited by in corpus (86)

• Transformers in Vision: A Survey
• Pre-trained Models for Natural Language Processing: A Survey
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• Towards artificial general intelligence via a multimodal foundation model
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• VLP: A Survey on Vision-Language Pre-training
• Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing
• Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion
• Learning Video Representations using Contrastive Bidirectional Transformer
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• Towards Natural Language Interfaces for Data Visualization: A Survey
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• Local-Global Context Aware Transformer for Language-Guided Video Segmentation
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Multimodal Few-Shot Learning with Frozen Language Models
• Disentangling Hate in Online Memes
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Efficient Self-supervised Vision Transformers for Representation Learning
• Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning
• ViGT: Proposal-free Video Grounding with Learnable Token in Transformer
• VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
• Context Disentangling and Prototype Inheriting for Robust Visual Grounding
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Reference-based Defect Detection Network
• Active Contrastive Learning of Audio-Visual Video Representations
• VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
• Parallel Attention Network with Sequence Matching for Video Grounding
• TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamese
• Universal Multimodal Representation for Language Understanding
• Robustness Verification for Transformers
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
• VD-BERT: A Unified Vision and Dialog Transformer with BERT
• Discrete and continuous representations and processing in deep learning: Looking forward
• Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation
• Visual Relationship Detection with Visual-Linguistic Knowledge from Multimodal Representations
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning
• Recommending Themes for Ad Creative Design via Visual-Linguistic Representations
• An Embarrassingly Simple Model for Dialogue Relation Extraction
• Knowledge Perceived Multi-modal Pretraining in E-commerce
• Masked Vision-Language Transformer in Fashion
• SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding
• Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
• Mobile App Crowdsourced Test Report Consistency Detection via Deep Image-and-Text Fusion Understanding
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• Cross-media Structured Common Space for Multimedia Event Extraction
• All-in-One Image-Grounded Conversational Agents
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
• Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
• Weakly supervised cross-domain alignment with optimal transport
• MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
• ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer
• An Empirical Study on the Language Modal in Visual Question Answering
• Visual Grounding Strategies for Text-Only Natural Language Processing
• Probing Contextualized Sentence Representations with Visual Awareness
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• Latent Alignment of Procedural Concepts in Multimodal Recipes
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Variational Knowledge Distillation for Disease Classification in Chest X-Rays
• HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
• Achieving Human Parity on Visual Question Answering
• Human-centric Relation Segmentation: Dataset and Solution
• Accurate Word Representations with Universal Visual Guidance
• T-EMDE: Sketching-based global similarity for cross-modal retrieval
• Panoptic Narrative Grounding
• NarrationBot and InfoBot: A Hybrid System for Automated Video Description
• Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructions
• Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
• Improving Calibration in Deep Metric Learning With Cross-Example Softmax