Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions

Abstract

Visual Question Answering (VQA) has attracted attention from both computer vision and natural language processing communities. Most existing approaches adopt the pipeline of representing an image via pre-trained CNNs, and then using the uninterpretable CNN features in conjunction with the question to predict the answer. Although such end-to-end models might report promising performance, they rarely provide any insight, apart from the answer, into the VQA process. In this work, we propose to break up the end-to-end VQA into two steps: explaining and reasoning, in an attempt towards a more explainable VQA by shedding light on the intermediate results between these two steps. To that end, we first extract attributes and generate descriptions as explanations for an image using pre-trained attribute detectors and image captioning models, respectively. Next, a reasoning module utilizes these explanations in place of the image to infer an answer to the question. The advantages of such a breakdown include: (1) the attributes and captions can reflect what the system extracts from the image, thus can provide some explanations for the predicted answer; (2) these intermediate results can help us identify the inabilities of both the image understanding part and the answer inference part when the predicted answer is wrong. We conduct extensive experiments on a popular VQA dataset and dissect all results according to several measurements of the explanation quality. Our system achieves comparable performance with the state-of-the-art, yet with added benefits of explainability and the inherent ability to further improve with higher quality explanations.

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Image Captioning with Semantic Attention
• Stacked Attention Networks for Image Question Answering
• Show and Tell: A Neural Image Caption Generator
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• A Focused Dynamic Attention Model for Visual Question Answering
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• From Captions to Visual Concepts and Back
• Dual Attention Networks for Multimodal Reasoning and Matching
• Visual7W: Grounded Question Answering in Images
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• What value do explicit high level concepts have in vision to language problems?

Cited by in corpus (8)

• A Fast and Accurate One-Stage Approach to Visual Grounding
• VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
• Joint Image Captioning and Question Answering
• Generating Question Relevant Captions to Aid Visual Question Answering
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• Non-monotonic Logical Reasoning Guiding Deep Learning for Explainable Visual Question Answering