Dual Attention Networks for Visual Reference Resolution in Visual Dialog

Abstract

Visual dialog (VisDial) is a task which requires an AI agent to answer a series of questions grounded in an image. Unlike in visual question answering (VQA), the series of questions should be able to capture a temporal context from a dialog history and exploit visually-grounded information. A problem called visual reference resolution involves these challenges, requiring the agent to resolve ambiguous references in a given question and find the references in a given image. In this paper, we propose Dual Attention Networks (DAN) for visual reference resolution. DAN consists of two kinds of attention networks, REFER and FIND. Specifically, REFER module learns latent relationships between a given question and a dialog history by employing a self-attention mechanism. FIND module takes image features and reference-aware representations (i.e., the output of REFER module) as input, and performs visual grounding via bottom-up attention mechanism. We qualitatively and quantitatively evaluate our model on VisDial v1.0 and v0.9 datasets, showing that DAN outperforms the previous state-of-the-art model by a significant margin.

arXiv commentEMNLP 2019

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Categorical Reparameterization with Gumbel-Softmax
• VQA: Visual Question Answering
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Reasoning about Entailment with Neural Attention
• Multimodal Residual Learning for Visual QA
• Visual Reference Resolution using Attention Memory for Visual Dialog
• Image-Question-Answer Synergistic Network for Visual Dialog
• Recursive Visual Attention in Visual Dialog

Cited by in corpus (6)

• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• Making History Matter: History-Advantage Sequence Training for Visual Dialog
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• Modality-Balanced Models for Visual Dialogue
• OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts