Information Maximizing Visual Question Generation

Abstract

Though image-to-sequence generation models have become overwhelmingly popular in human-computer communications, they suffer from strongly favoring safe generic questions ("What is in this picture?"). Generating uninformative but relevant questions is not sufficient or useful. We argue that a good question is one that has a tightly focused purpose --- one that is aimed at expecting a specific type of response. We build a model that maximizes mutual information between the image, the expected answer and the generated question. To overcome the non-differentiability of discrete natural language tokens, we introduce a variational continuous latent space onto which the expected answers project. We regularize this latent space with a second latent space that ensures clustering of similar answers. Even when we don't know the expected answer, this second latent space can generate goal-driven questions specifically aimed at extracting objects ("what is the person throwing"), attributes, ("What kind of shirt is the person wearing?"), color ("what color is the frisbee?"), material ("What material is the frisbee?"), etc. We quantitatively show that our model is able to retain information about an expected answer category, resulting in more diverse, goal-driven questions. We launch our model on a set of real world images and extract previously unseen visual concepts.

CVPR 2019

References in corpus (24)

• Auto-Encoding Variational Bayes
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Categorical Reparameterization with Gumbel-Softmax
• Neural Discrete Representation Learning
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Semi-Supervised Learning with Deep Generative Models
• A Deep Reinforced Model for Abstractive Summarization
• VQA: Visual Question Answering
• DRAW: A Recurrent Neural Network For Image Generation
• A Recurrent Latent Variable Model for Sequential Data
• Dual Learning for Machine Translation
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Exploring Models and Data for Image Question Answering
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Deep Kalman Filters
• Black box variational inference for state space models
• Learning to Ask: Neural Question Generation for Reading Comprehension
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• A Joint Model for Question Answering and Question Generation
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Visual Question Generation as Dual Task of Visual Question Answering
• Automatic Generation of Grounded Visual Questions
• Visual Curiosity: Learning to Ask Questions to Learn Visual Recognition