Reasoning Visual Dialogs with Structural and Partial Observations

Abstract

We propose a novel model to address the task of Visual Dialog which exhibits complex dialog structures. To obtain a reasonable answer based on the current question and the dialog history, the underlying semantic dependencies between dialog entities are essential. In this paper, we explicitly formalize this task as inference in a graphical model with partially observed nodes and unknown graph structures (relations in dialog). The given dialog entities are viewed as the observed nodes. The answer to a given question is represented by a node with missing value. We first introduce an Expectation Maximization algorithm to infer both the underlying dialog structures and the missing node values (desired answers). Based on this, we proceed to propose a differentiable graph neural network (GNN) solution that approximates this process. Experiment results on the VisDial and VisDial-Q datasets show that our model outperforms comparative methods. It is also observed that our method can infer the underlying dialog structure for better dialog reasoning.

CVPR 2019 Oral paper

References in corpus (12)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Neural Message Passing for Quantum Chemistry
• Conditional Random Fields as Recurrent Neural Networks
• Interaction Networks for Learning about Objects, Relations and Physics
• Exploring Models and Data for Image Question Answering
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• Visual Reference Resolution using Attention Memory for Visual Dialog
• Discriminability objective for training descriptive captions
• Geometric deep learning on graphs and manifolds using mixture model CNNs
• CRF-CNN: Modeling Structured Information in Human Pose Estimation
• Deeply Learning the Messages in Message Passing Inference
• FlipDial: A Generative Model for Two-Way Visual Dialogue

Cited by in corpus (11)

• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
• Hierarchical Human Parsing with Typed Part-Relation Reasoning
• Cascaded Human-Object Interaction Recognition
• DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
• ORD: Object Relationship Discovery for Visual Dialogue Generation
• Referring Image Segmentation via Cross-Modal Progressive Comprehension
• Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• Guessing State Tracking for Visual Dialogue
• Proposal-free One-stage Referring Expression via Grid-Word Cross-Attention