Ask Your Neurons: A Neural-based Approach to Answering Questions about Images

Abstract

We address a question answering task on real-world images that is set up as a Visual Turing Test. By combining latest advances in image representation and natural language processing, we propose Neural-Image-QA, an end-to-end formulation to this problem for which all parts are trained jointly. In contrast to previous efforts, we are facing a multi-modal problem where the language output (answer) is conditioned on visual and natural language input (image and question). Our approach Neural-Image-QA doubles the performance of the previous best approach on this problem. We provide additional insights into the problem by analyzing how much information is contained only in the language part for which we provide a new human baseline. To study human consensus, which is related to the ambiguities inherent in this challenging task, we propose two novel metrics and collect additional answers which extends the original DAQUAR dataset to DAQUAR-Consensus.

ICCV'15 (Oral)

Cited by in corpus (88)

• Recent Trends in Deep Learning Based Natural Language Processing
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Modulating early visual processing by language
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Multimodal Residual Learning for Visual QA
• Multimodal Machine Learning: A Survey and Taxonomy
• A Focused Dynamic Attention Model for Visual Question Answering
• Generation and Comprehension of Unambiguous Object Descriptions
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Explicit Knowledge-based Reasoning for Visual Question Answering
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Visual Reference Resolution using Attention Memory for Visual Dialog
• Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
• Learning Language-Visual Embedding for Movie Understanding with Natural-Language
• Visually Grounded Neural Syntax Acquisition
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Person Search with Natural Language Description
• Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• High-Order Attention Models for Visual Question Answering
• Visual Question Answering: A Survey of Methods and Datasets
• Visual7W: Grounded Question Answering in Images
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Yin and Yang: Balancing and Answering Binary Visual Questions
• FVQA: Fact-based Visual Question Answering
• Compositional Memory for Visual Question Answering
• TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
• What value do explicit high level concepts have in vision to language problems?
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Revisiting Visual Question Answering Baselines
• Visual Question Generation as Dual Task of Visual Question Answering
• MemexQA: Visual Memex Question Answering
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Task-driven Visual Saliency and Attention-based Visual Question Answering
• An Analysis of Visual Question Answering Algorithms
• Active Learning for Visual Question Answering: An Empirical Study
• Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
• Neural Self Talk: Image Understanding via Continuous Questioning and Answering
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• Robustness Analysis of Visual QA Models by Basic Questions
• Finding Answers from the Word of God: Domain Adaptation for Neural Networks in Biblical Question Answering
• Visual Question Answering with Memory-Augmented Networks
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• Joint Image Captioning and Question Answering
• ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
• Learning Visual Storylines with Skipping Recurrent Neural Networks
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• Learning Visual Question Answering by Bootstrapping Hard Attention
• Fooling Vision and Language Models Despite Localization and Attention Mechanism
• Dense Captioning with Joint Inference and Visual Context
• Textually Enriched Neural Module Networks for Visual Question Answering
• Deep Binaries: Encoding Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval
• Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
• Category-Based Deep CCA for Fine-Grained Venue Discovery from Multimodal Data
• Learning to Disambiguate by Asking Discriminative Questions
• AlignVE: Visual Entailment Recognition Based on Alignment Relations
• Visual Question: Predicting If a Crowd Will Agree on the Answer
• Answering Image Riddles using Vision and Reasoning through Probabilistic Soft Logic
• Spatial Memory for Context Reasoning in Object Detection
• Examining Cooperation in Visual Dialog Models
• Ladder Loss for Coherent Visual-Semantic Embedding
• Paying Attention to Descriptions Generated by Image Captioning Models
• Question Guided Modular Routing Networks for Visual Question Answering
• Tutorial on Answering Questions about Images with Deep Learning
• Evaluating Text-to-Image Matching using Binary Image Selection (BISON)
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
• Leveraging Visual Question Answering for Image-Caption Ranking
• Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"
• Holistic Multi-modal Memory Network for Movie Question Answering
• The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
• cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey
• Pay Attention to Those Sets! Learning Quantification from Images
• Exploration on Grounded Word Embedding: Matching Words and Images with Image-Enhanced Skip-Gram Model
• Recurrent and Contextual Models for Visual Question Answering
• Semantic bottleneck for computer vision tasks
• Contrastive Learning of Visual-Semantic Embeddings
• Middle-Out Decoding
• Eliminating Catastrophic Interference with Biased Competition