UNITER: UNiversal Image-TExt Representation Learning

Abstract

Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are simultaneously processed for joint visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design four pre-training tasks: Masked Language Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text Matching (ITM), and Word-Region Alignment (WRA). Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). In addition to ITM for global image-text alignment, we also propose WRA via the use of Optimal Transport (OT) to explicitly encourage fine-grained alignment between words and image regions during pre-training. Comprehensive analysis shows that both conditional masking and OT-based WRA contribute to better pre-training. We also conduct a thorough ablation study to find an optimal combination of pre-training tasks. Extensive experiments show that UNITER achieves new state of the art across six V+L tasks (over nine datasets), including Visual Question Answering, Image-Text Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR${}^2$. Code is available at https://github.com/ChenRocks/UNITER.

arXiv commentECCV 2020

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Learning Video Representations using Contrastive Bidirectional Transformer
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Selfie: Self-supervised Pretraining for Image Embedding
• Graph Optimal Transport for Cross-Domain Alignment
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• NLVR2 Visual Bias Analysis

Cited by in corpus (62)

• Learning Transferable Visual Models From Natural Language Supervision
• Language Models are Few-Shot Learners
• Pre-trained Models for Natural Language Processing: A Survey
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• MERLOT: Multimodal Neural Script Knowledge Models
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Modality-Agnostic Attention Fusion for visual search with text feedback
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• Unshuffling Data for Improved Generalization
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• Support-set bottlenecks for video-text representation learning
• A Multimodal Memes Classification: A Survey and Open Research Issues
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog
• SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
• In Defense of Grid Features for Visual Question Answering
• Contrastive Visual-Linguistic Pretraining
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
• Meta Module Network for Compositional Visual Reasoning
• Deep Multimodal Neural Architecture Search
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
• Cross-media Structured Common Space for Multimedia Event Extraction
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
• What Makes A Good Story? Designing Composite Rewards for Visual Storytelling
• All-in-One Image-Grounded Conversational Agents
• Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
• KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for Visual Commonsense Reasoning
• Learning Object Detection from Captions via Textual Scene Attributes
• Weakly supervised cross-domain alignment with optimal transport
• Pre-training Graph Transformer with Multimodal Side Information for Recommendation
• Analogical Reasoning for Visually Grounded Language Acquisition
• Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• Visually Grounded Continual Learning of Compositional Phrases
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• Variational Knowledge Distillation for Disease Classification in Chest X-Rays
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Human-centric Relation Segmentation: Dataset and Solution
• ActBERT: Learning Global-Local Video-Text Representations
• SeqDialN: Sequential Visual Dialog Networks in Joint Visual-Linguistic Representation Space
• Pre-training for Video Captioning Challenge 2020 Summary
• Analysis on Image Set Visual Question Answering
• GEM: A General Evaluation Benchmark for Multimodal Tasks
• Utilizing Every Image Object for Semi-supervised Phrase Grounding
• Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training
• Comparing Visual Reasoning in Humans and AI