Factor Graph Attention

Abstract

Dialog is an effective way to exchange information, but subtle details and nuances are extremely important. While significant progress has paved a path to address visual dialog with algorithms, details and nuances remain a challenge. Attention mechanisms have demonstrated compelling results to extract details in visual question answering and also provide a convincing framework for visual dialog due to their interpretability and effectiveness. However, the many data utilities that accompany visual dialog challenge existing attention techniques. We address this issue and develop a general attention mechanism for visual dialog which operates on any number of data utilities. To this end, we design a factor graph based attention mechanism which combines any number of utility representations. We illustrate the applicability of the proposed approach on the challenging and recently introduced VisDial datasets, outperforming recent state-of-the-art methods by 1.1% for VisDial0.9 and by 2% for VisDial1.0 on MRR. Our ensemble model improved the MRR score on VisDial1.0 by more than 6%.

Accepted to CVPR 2019; revised version includes bottom-up features

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• Hadamard Product for Low-rank Bilinear Pooling
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Visual Reference Resolution using Attention Memory for Visual Dialog
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Visual Question Generation as Dual Task of Visual Question Answering
• Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning

Cited by in corpus (7)

• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
• Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
• Vision-Dialog Navigation by Exploring Cross-modal Memory
• KBGN: Knowledge-Bridge Graph Network for Adaptive Vision-Text Reasoning in Visual Dialogue
• GoG: Relation-aware Graph-over-Graph Network for Visual Dialog