Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System

Abstract

Understanding dynamic scenes and dialogue contexts in order to converse with users has been challenging for multimodal dialogue systems. The 8-th Dialog System Technology Challenge (DSTC8) proposed an Audio Visual Scene-Aware Dialog (AVSD) task, which contains multiple modalities including audio, vision, and language, to evaluate how dialogue systems understand different modalities and response to users. In this paper, we proposed a multi-step joint-modality attention network (JMAN) based on recurrent neural network (RNN) to reason on videos. Our model performs a multi-step attention mechanism and jointly considers both visual and textual representations in each reasoning process to better integrate information from the two different modalities. Compared to the baseline released by AVSD organizers, our model achieves a relative 12.1% and 22.4% improvement over the baseline on ROUGE-L score and CIDEr score.

DSTC8 collocated with AAAI2020

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Stacked Attention Networks for Image Question Answering
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Video Question Answering via Attribute-Augmented Attention Network Learning
• Visual Reference Resolution using Attention Memory for Visual Dialog
• Bilinear Attention Networks
• TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
• DeepStory: Video Story QA by Deep Embedded Memory Networks
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
• A Read-Write Memory Network for Movie Story Understanding
• End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
• Recursive Visual Attention in Visual Dialog

Cited by in corpus (5)

• Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue
• Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues
• HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue
• $C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues
• VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks