Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
arXiv:1711.07613
Abstract
The Visual Dialogue task requires an agent to engage in a conversation about an image with a human. It represents an extension of the Visual Question Answering task in that the agent needs to answer a question about an image, but it needs to do so in light of the previous dialogue that has taken place. The key challenge in Visual Dialogue is thus maintaining a consistent, and natural dialogue while continuing to answer questions correctly. We present a novel approach that combines Reinforcement Learning and Generative Adversarial Networks (GANs) to generate more human-like responses to questions. The GAN helps overcome the relative paucity of training data, and the tendency of the typical MLE-based approach to generate overly terse answers. Critically, the GAN is tightly integrated into the attention mechanism that generates human-interpretable reasons for each answer. This means that the discriminative model of the GAN has the task of assessing whether a candidate answer is generated by a human or not, given the provided reason. This is significant because it drives the generative model to produce high quality answers that are well supported by the associated reasoning. The method also generates the state-of-the-art results on the primary benchmark.
References in corpus (29)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
- A Neural Conversational Model
- Improved Techniques for Training GANs
- InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Deep Reinforcement Learning for Dialogue Generation
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
- Adversarial Learning for Neural Dialogue Generation
- Stacked Attention Networks for Image Question Answering
- Show and Tell: A Neural Image Caption Generator
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
- Towards Diverse and Natural Image Descriptions via a Conditional GAN
- Continuously Learning Neural Dialogue Management
- Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
- Visual Reference Resolution using Attention Memory for Visual Dialog
- Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
- A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
- Modeling Context in Referring Expressions
- Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
- What value do explicit high level concepts have in vision to language problems?
- Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
- Natural Language Object Retrieval
- Deep Active Learning for Dialogue Generation