Learning to Disambiguate by Asking Discriminative Questions
arXiv:1708.02760
Abstract
The ability to ask questions is a powerful tool to gather information in order to learn about the world and resolve ambiguities. In this paper, we explore a novel problem of generating discriminative questions to help disambiguate visual instances. Our work can be seen as a complement and new extension to the rich research studies on image captioning and question answering. We introduce the first large-scale dataset with over 10,000 carefully annotated images-question tuples to facilitate benchmarking. In particular, each tuple consists of a pair of images and 4.6 discriminative questions (as positive samples) and 5.9 non-discriminative questions (as negative samples) on average. In addition, we present an effective method for visual discriminative question generation. The method can be trained in a weakly supervised manner without discriminative images-question tuples but just existing visual question answering datasets. Promising results are shown against representative baselines through quantitative evaluations and user studies.
arXiv comment14 pages, 12 figures, ICCV2017
References in corpus (25)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- LSTM: A Search Space Odyssey
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Stacked Attention Networks for Image Question Answering
- Show and Tell: A Neural Image Caption Generator
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
- Generation and Comprehension of Unambiguous Object Descriptions
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
- deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
- From Captions to Visual Concepts and Back
- Generating Visual Explanations
- Modeling Context in Referring Expressions
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- Visual7W: Grounded Question Answering in Images
- Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
- Yin and Yang: Balancing and Answering Binary Visual Questions
- Natural Language Object Retrieval
- GuessWhat?! Visual object discovery through multi-modal dialogue
- Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
- A Hierarchical Approach for Generating Descriptive Image Paragraphs