The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions

Abstract

One of the most intriguing features of the Visual Question Answering (VQA) challenge is the unpredictability of the questions. Extracting the information required to answer them demands a variety of image operations from detection and counting, to segmentation and reconstruction. To train a method to perform even one of these operations accurately from {image,question,answer} tuples would be challenging, but to aim to achieve them all with a limited set of such training data seems ambitious at best. We propose here instead a more general and scalable approach which exploits the fact that very good methods to achieve these operations already exist, and thus do not need to be trained. Our method thus learns how to exploit a set of external off-the-shelf algorithms to achieve its goal, an approach that has something in common with the Neural Turing Machine. The core of our proposed method is a new co-attention model. In addition, the proposed approach generates human-readable reasons for its decision, and can still be trained end-to-end without ground truth reasons being given. We demonstrate the effectiveness on two publicly available datasets, Visual Genome and VQA, and show that it produces the state-of-the-art results in both cases.

References in corpus (24)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• Going Deeper with Convolutions
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Verb Semantics and Lexical Selection
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Simple Baseline for Visual Question Answering
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Multimodal Residual Learning for Visual QA
• Stacked Attention Networks for Image Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• A Focused Dynamic Attention Model for Visual Question Answering
• Learning to Compose Neural Networks for Question Answering
• Visual Relationship Detection with Language Priors
• Explicit Knowledge-based Reasoning for Visual Question Answering
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Visual Question Answering: A Survey of Methods and Datasets
• Visual7W: Grounded Question Answering in Images
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• FVQA: Fact-based Visual Question Answering
• What value do explicit high level concepts have in vision to language problems?
• DualNet: Domain-Invariant Network for Visual Question Answering