Image-Question-Answer Synergistic Network for Visual Dialog

Abstract

The image, question (combined with the history for de-referencing), and the corresponding answer are three vital components of visual dialog. Classical visual dialog systems integrate the image, question, and history to search for or generate the best matched answer, and so, this approach significantly ignores the role of the answer. In this paper, we devise a novel image-question-answer synergistic network to value the role of the answer for precise visual dialog. We extend the traditional one-stage solution to a two-stage solution. In the first stage, candidate answers are coarsely scored according to their relevance to the image and question pair. Afterward, in the second stage, answers with high probability of being correct are re-ranked by synergizing with image and question. On the Visual Dialog v1.0 dataset, the proposed synergistic network boosts the discriminative visual dialog model to achieve a new state-of-the-art of 57.88\% normalized discounted cumulative gain. A generative visual dialog model equipped with the proposed technique also shows promising improvements.

Accepted by cvpr2019

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Categorical Reparameterization with Gumbel-Softmax
• VQA: Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• A Theoretical Analysis of NDCG Type Ranking Measures
• FiLM: Visual Reasoning with a General Conditioning Layer
• Bilinear Attention Networks
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Dual Attention Networks for Multimodal Reasoning and Matching
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning

Cited by in corpus (5)

• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
• Dual Attention Networks for Visual Reference Resolution in Visual Dialog
• DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
• Modality-Balanced Models for Visual Dialogue
• Recursive Visual Attention in Visual Dialog