Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
arXiv:2108.09479
Abstract
Existing approaches to vision-language pre-training (VLP) heavily rely on an object detector based on bounding boxes (regions), where salient objects are first detected from images and then a Transformer-based model is used for cross-modal fusion. Despite their superior performance, these approaches are bounded by the capability of the object detector in terms of both effectiveness and efficiency. Besides, the presence of object detection imposes unnecessary constraints on model designs and makes it difficult to support end-to-end training. In this paper, we revisit grid-based convolutional features for vision-language pre-training, skipping the expensive region-related steps. We propose a simple yet effective grid-based VLP method that works surprisingly well with the grid features. By pre-training only with in-domain datasets, the proposed Grid-VLP method can outperform most competitive region-based VLP methods on three examined vision-language understanding tasks. We hope that our findings help to further advance the state of the art of vision-language pre-training, and provide a new direction towards effective and efficient VLP.
References in corpus (8)
- VQA: Visual Question Answering
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- UNITER: UNiversal Image-TExt Representation Learning
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
- UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
- In Defense of Grid Features for Visual Question Answering