Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog

Abstract

This paper presents a new model for visual dialog, Recurrent Dual Attention Network (ReDAN), using multi-step reasoning to answer a series of questions about an image. In each question-answering turn of a dialog, ReDAN infers the answer progressively through multiple reasoning steps. In each step of the reasoning process, the semantic representation of the question is updated based on the image and the previous dialog history, and the recurrently-refined representation is used for further reasoning in the subsequent step. On the VisDial v1.0 dataset, the proposed ReDAN model achieves a new state-of-the-art of 64.47% NDCG score. Visualization on the reasoning process further demonstrates that ReDAN can locate context-relevant visual and textual clues via iterative refinement, which can lead to the correct answer step-by-step.

Accepted to ACL 2019

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• VQA: Visual Question Answering
• ReasoNet: Learning to Stop Reading in Machine Comprehension
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Hadamard Product for Low-rank Bilinear Pooling
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• Visual Reference Resolution using Attention Memory for Visual Dialog
• Relation-Aware Graph Attention Network for Visual Question Answering
• Evaluating Visual Conversational Agents via Cooperative Human-AI Games
• Reasoning Visual Dialogs with Structural and Partial Observations
• FlipDial: A Generative Model for Two-Way Visual Dialogue
• Factor Graph Attention
• Dual Attention Networks for Visual Reference Resolution in Visual Dialog
• Making History Matter: History-Advantage Sequence Training for Visual Dialog
• Image-Question-Answer Synergistic Network for Visual Dialog
• Generative Visual Dialogue System via Adaptive Reasoning and Weighted Likelihood Estimation
• Recursive Visual Attention in Visual Dialog

Cited by in corpus (10)

• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
• Generative Visual Dialogue System via Adaptive Reasoning and Weighted Likelihood Estimation
• Modality-Balanced Models for Visual Dialogue
• DTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• Saying the Unseen: Video Descriptions via Dialog Agents