Learning to Count Objects in Natural Images for Visual Question Answering

Abstract

Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-the-art accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.

Published in ICLR 2018

References in corpus (9)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• A simple neural network module for relational reasoning
• Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Interpretable Counting for Visual Question Answering
• Learning non-maximum suppression
• Structured Attentions for Visual Question Answering
• Learning Detection with Diverse Proposals

Cited by in corpus (30)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• RSVQA: Visual Question Answering for Remote Sensing Data
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data
• Bilinear Attention Networks
• Relation-Aware Graph Attention Network for Visual Question Answering
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Multi-modality Latent Interaction Network for Visual Question Answering
• Multimodal Unified Attention Networks for Vision-and-Language Interactions
• Learning to Compose Dynamic Tree Structures for Visual Contexts
• Auto-Encoding Scene Graphs for Image Captioning
• Counterfactual Critic Multi-Agent Training for Scene Graph Generation
• In Defense of Grid Features for Visual Question Answering
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• Linguistically-aware Attention for Reducing the Semantic-Gap in Vision-Language Tasks
• Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
• GPS-Net: Graph Property Sensing Network for Scene Graph Generation
• Dual ResGCN for Balanced Scene GraphGeneration
• Interpretable Visual Question Answering by Reasoning on Dependency Trees
• Bilinear Graph Networks for Visual Question Answering
• TallyQA: Answering Complex Counting Questions
• AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss
• Visual Question Answering based on Local-Scene-Aware Referring Expression Generation
• Uncertainty based Class Activation Maps for Visual Question Answering
• Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering
• Analysis on Image Set Visual Question Answering
• Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation
• Visual Question Answering Using Semantic Information from Image Descriptions