Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Abstract

We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM and Unicoder, both visual and linguistic contents are fed into a multi-layer Transformer for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling (MLM), Masked Object Classification (MOC) and Visual-linguistic Matching (VLM). The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly. The last task tries to predict whether an image and a text describe each other. After pretraining on large-scale image-caption pairs, we transfer Unicoder-VL to caption-based image-text retrieval and visual commonsense reasoning, with just one additional output layer. We achieve state-of-the-art or comparable results on both two tasks and show the powerful ability of the cross-modal pre-training.

accepted by AAAI-2020. arXiv admin note: text overlap with arXiv:1909.11740 by other authors

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Cross-lingual Language Model Pretraining
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• A large annotated corpus for learning natural language inference
• UNITER: UNiversal Image-TExt Representation Learning
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Cited by in corpus (55)

• Learning Transferable Visual Models From Natural Language Supervision
• Transformers in Vision: A Survey
• Pre-trained Models for Natural Language Processing: A Survey
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Learning Video Representations using Contrastive Bidirectional Transformer
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• UNITER: UNiversal Image-TExt Representation Learning
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Efficient Self-supervised Vision Transformers for Representation Learning
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• M6: A Chinese Multimodal Pretrainer
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
• Unshuffling Data for Improved Generalization
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• Discrete and continuous representations and processing in deep learning: Looking forward
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• Learning the Best Pooling Strategy for Visual Semantic Embedding
• Recommending Themes for Ad Creative Design via Visual-Linguistic Representations
• OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
• Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
• Scaling Up Vision-Language Pre-training for Image Captioning
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
• Cross-media Structured Common Space for Multimedia Event Extraction
• All-in-One Image-Grounded Conversational Agents
• KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for Visual Commonsense Reasoning
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Weakly supervised cross-domain alignment with optimal transport
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• LAMP: Label Augmented Multimodal Pretraining
• Learning to Represent Image and Text with Denotation Graph
• Probing Contextualized Sentence Representations with Visual Awareness
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• SeqDialN: Sequential Visual Dialog Networks in Joint Visual-Linguistic Representation Space
• ActBERT: Learning Global-Local Video-Text Representations
• Achieving Human Parity on Visual Question Answering
• Team RUC_AIM3 Technical Report at ActivityNet 2021: Entities Object Localization
• Separating Skills and Concepts for Novel Visual Question Answering
• PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling
• Improving Calibration in Deep Metric Learning With Cross-Example Softmax
• Accurate Word Representations with Universal Visual Guidance