ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph

Abstract

We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. ERNIE-ViL tries to build the detailed semantic connections (objects, attributes of objects and relationships between objects) across vision and language, which are essential to vision-language cross-modal tasks. Utilizing scene graphs of visual scenes, ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction tasks in the pre-training phase. Specifically, these prediction tasks are implemented by predicting nodes of different types in the scene graph parsed from the sentence. Thus, ERNIE-ViL can learn the joint representations characterizing the alignments of the detailed semantics across vision and language. After pre-training on large scale image-text aligned datasets, we validate the effectiveness of ERNIE-ViL on 5 cross-modal downstream tasks. ERNIE-ViL achieves state-of-the-art performances on all these tasks and ranks the first place on the VCR leaderboard with an absolute improvement of 3.7%.

Paper has been published in the AAAI2021 conference

References in corpus (10)

• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• ERNIE: Enhanced Representation through Knowledge Integration
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• UNITER: UNiversal Image-TExt Representation Learning
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Circle Loss: A Unified Perspective of Pair Similarity Optimization
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering

Cited by in corpus (38)

• Learning Transferable Visual Models From Natural Language Supervision
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Florence: A New Foundation Model for Computer Vision
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• CLIPort: What and Where Pathways for Robotic Manipulation
• Disentangling Hate in Online Memes
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• Referring Transformer: A One-step Approach to Multi-task Visual Grounding
• VinVL: Revisiting Visual Representations in Vision-Language Models
• InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
• MERLOT: Multimodal Neural Script Knowledge Models
• M6: A Chinese Multimodal Pretrainer
• A Multimodal Framework for the Detection of Hateful Memes
• Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
• Scaling Up Vision-Language Pre-training for Image Captioning
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• M6-T: Exploring Sparse Expert Models and Beyond
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding
• Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
• KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for Visual Commonsense Reasoning
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Causal Attention for Vision-Language Tasks
• GlyphCRM: Bidirectional Encoder Representation for Chinese Character with its Glyph
• ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration
• Interpretable Visual Understanding with Cognitive Attention Network
• Achieving Human Parity on Visual Question Answering
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
• Learning Relation Alignment for Calibrated Cross-modal Retrieval
• A Transformer-based Cross-modal Fusion Model with Adversarial Training for VQA Challenge 2021
• Domain-robust VQA with diverse datasets and methods but no target labels