In Defense of Grid Features for Visual Question Answering

Abstract

Popularized as 'bottom-up' attention, bounding box (or region) based visual features have recently surpassed vanilla grid-based convolutional features as the de facto standard for vision and language tasks like visual question answering (VQA). However, it is not clear whether the advantages of regions (e.g. better localization) are the key reasons for the success of bottom-up attention. In this paper, we revisit grid features for VQA, and find they can work surprisingly well - running more than an order of magnitude faster with the same accuracy (e.g. if pre-trained in a similar fashion). Through extensive experiments, we verify that this observation holds true across different VQA models (reporting a state-of-the-art accuracy on VQA 2.0 test-std, 72.71), datasets, and generalizes well to other tasks like image captioning. As grid features make the model design and training process much simpler, this enables us to train them end-to-end and also use a more flexible network design. We learn VQA models end-to-end, from pixels directly to answers, and show that strong performance is achievable without using any region annotations in pre-training. We hope our findings help further improve the scientific understanding and the practical application of VQA. Code and features will be made available.

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• YFCC100M: The New Data in Multimedia Research
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• A simple neural network module for relational reasoning
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Learning to Count Objects in Natural Images for Visual Question Answering
• UNITER: UNiversal Image-TExt Representation Learning
• Exploring Nearest Neighbor Approaches for Image Captioning
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Interpretable Counting for Visual Question Answering

Cited by in corpus (7)

• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
• WebQA: Multihop and Multimodal QA
• Visual Question Answering Using Semantic Information from Image Descriptions