An Empirical Evaluation of Visual Question Answering for Novel Objects
arXiv:1704.02516
Abstract
We study the problem of answering questions about images in the harder setting, where the test questions and corresponding images contain novel objects, which were not queried about in the training data. Such setting is inevitable in real world-owing to the heavy tailed distribution of the visual categories, there would be some objects which would not be annotated in the train set. We show that the performance of two popular existing methods drop significantly (up to 28%) when evaluated on novel objects cf. known objects. We propose methods which use large existing external corpora of (i) unlabeled text, i.e. books, and (ii) images tagged with classes, to achieve novel object based visual question answering. We do systematic empirical studies, for both an oracle case where the novel objects are known textually, as well as a fully automatic case without any explicit knowledge of the novel objects, but with the minimal assumption that the novel objects are semantically related to the existing objects in training. The proposed methods for novel object based visual question answering are modular and can potentially be used with many visual question answering architectures. We show consistent improvements with the two popular architectures and give qualitative analysis of the cases where the model does well and of those where it fails to bring improvements.
arXiv comment11 pages, 4 figures, accepted in CVPR 2017 (poster)
References in corpus (18)
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- A Hierarchical Neural Autoencoder for Paragraphs and Documents
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Learning to Compose Neural Networks for Question Answering
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
- Explicit Knowledge-based Reasoning for Visual Question Answering
- Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
- Visual Question Answering: A Survey of Methods and Datasets
- Visual7W: Grounded Question Answering in Images
- Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
- FVQA: Fact-based Visual Question Answering
- Learning Hypergraph-regularized Attribute Predictors