Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
arXiv:2111.08276
Abstract
Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.
arXiv commentICML 2022
References in corpus (10)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Decoupled Weight Decay Regularization
- Learning Transferable Visual Models From Natural Language Supervision
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Unifying Vision-and-Language Tasks via Text Generation