Zero-Shot Visual Question Answering
arXiv:1611.05546
Abstract
Part of the appeal of Visual Question Answering (VQA) is its promise to answer new questions about previously unseen images. Most current methods demand training questions that illustrate every possible concept, and will therefore never achieve this capability, since the volume of required training data would be prohibitive. Answering general questions about images requires methods capable of Zero-Shot VQA, that is, methods able to answer questions beyond the scope of the training questions. We propose a new evaluation protocol for VQA methods which measures their ability to perform Zero-Shot VQA, and in doing so highlights significant practical deficiencies of current approaches, some of which are masked by the biases in current datasets. We propose and evaluate several strategies for achieving Zero-Shot VQA, including methods based on pretrained word embeddings, object classifiers with semantic embeddings, and test-time retrieval of example images. Our extensive experiments are intended to serve as baselines for Zero-Shot VQA, and they also achieve state-of-the-art performance in the standard VQA evaluation setting.
References in corpus (10)
- ADADELTA: An Adaptive Learning Rate Method
- Matching Networks for One Shot Learning
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Simple Baseline for Visual Question Answering
- Explicit Knowledge-based Reasoning for Visual Question Answering
- Visual Question Answering: A Survey of Methods and Datasets
- Yin and Yang: Balancing and Answering Binary Visual Questions
- Revisiting Visual Question Answering Baselines
- Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
- Hierarchical Memory Networks for Answer Selection on Unknown Words
Cited by in corpus (11)
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
- C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
- Compact Trilinear Interaction for Visual Question Answering
- Zero-Shot Transfer VQA Dataset
- Zero-shot Visual Question Answering using Knowledge Graph
- Knowledge-aware Zero-Shot Learning: Survey and Perspective
- Visual Question Answering with Prior Class Semantics
- Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
- Separating Skills and Concepts for Novel Visual Question Answering
- Non-monotonic Logical Reasoning Guiding Deep Learning for Explainable Visual Question Answering