Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
arXiv:2106.13488
Abstract
Vision-Language Pre-training (VLP) aims to learn multi-modal representations from image-text pairs and serves for downstream vision-language tasks in a fine-tuning fashion. The dominant VLP models adopt a CNN-Transformer architecture, which embeds images with a CNN, and then aligns images and text with a Transformer. Visual relationship between visual contents plays an important role in image understanding and is the basic for inter-modal alignment learning. However, CNNs have limitations in visual relation learning due to local receptive field's weakness in modeling long-range dependencies. Thus the two objectives of learning visual relation and inter-modal alignment are encapsulated in the same Transformer network. Such design might restrict the inter-modal alignment learning in the Transformer by ignoring the specialized characteristic of each objective. To tackle this, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment. Specifically, we propose a metric named Inter-Modality Flow (IMF) to measure the interaction between vision and language modalities (i.e., inter-modality). We also design a novel masking optimization mechanism named Masked Feature Regression (MFR) in Transformer to further promote the inter-modality learning. To the best of our knowledge, this is the first study to explore the benefit of Transformer for visual feature learning in VLP. We verify our method on a wide range of vision-language tasks, including Image-Text Retrieval, Visual Question Answering (VQA), Visual Entailment and Visual Reasoning. Our approach not only outperforms the state-of-the-art VLP performance, but also shows benefits on the IMF metric.
Accepted by NeurIPS 2021
References in corpus (12)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Deformable DETR: Deformable Transformers for End-to-End Object Detection
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Toward Transformer-Based Object Detection
- A Fast Proximal Point Method for Computing Exact Wasserstein Distance
- Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
- e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations
- Visual Entailment Task for Visually-Grounded Language Learning