SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

Abstract

Vision-language pre-training (VLP) on large-scale image-text pairs has recently witnessed rapid progress for learning cross-modal representations. Existing pre-training methods either directly concatenate image representation and text representation at a feature level as input to a single-stream Transformer, or use a two-stream cross-modal Transformer to align the image-text representation at a high-level semantic space. In real-world image-text data, we observe that it is easy for some of the image-text pairs to align simple semantics on both modalities, while others may be related after higher-level abstraction. Therefore, in this paper, we propose a new pre-training method SemVLP, which jointly aligns both the low-level and high-level semantics between image and text representations. The model is pre-trained iteratively with two prevalent fashions: single-stream pre-training to align at a fine-grained feature level and two-stream pre-training to align high-level semantics, by employing a shared Transformer network with a pluggable cross-modal attention module. An extensive set of experiments have been conducted on four well-established vision-language understanding tasks to demonstrate the effectiveness of the proposed SemVLP in aligning cross-modal representations towards different semantic granularities.

10 pages, 4 figures

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• VQA: Visual Question Answering
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• UNITER: UNiversal Image-TExt Representation Learning
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Meta Module Network for Compositional Visual Reasoning

Cited by in corpus (5)

• What Vision-Language Models `See' when they See Scenes
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• Causal Attention for Vision-Language Tasks
• Achieving Human Parity on Visual Question Answering
• Data Efficient Masked Language Modeling for Vision and Language