Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions

Abstract

Visual Question Answering is a multi-modal task that aims to measure high-level visual understanding. Contemporary VQA models are restrictive in the sense that answers are obtained via classification over a limited vocabulary (in the case of open-ended VQA), or via classification over a set of multiple-choice-type answers. In this work, we present a completely generative formulation where a multi-word answer is generated for a visual query. To take this a step forward, we introduce a new task: ViQAR (Visual Question Answering and Reasoning), wherein a model must generate the complete answer and a rationale that seeks to justify the generated answer. We propose an end-to-end architecture to solve this task and describe how to evaluate it. We show that our model generates strong answers and rationales through qualitative and quantitative evaluation, as well as through a human Turing Test.

MULA Workshop, CVPR 2021

References in corpus (13)

• BERTScore: Evaluating Text Generation with BERT
• Skip-Thought Vectors
• A simple neural network module for relational reasoning
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Image Captioning with Semantic Attention
• Hadamard Product for Low-rank Bilinear Pooling
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
• Reasoning Visual Dialogs with Structural and Partial Observations
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions