Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modeling

Abstract

Visual question answering and visual dialogue tasks have been increasingly studied in the multimodal field towards more practical real-world scenarios. A more challenging task, audio visual scene-aware dialogue (AVSD), is proposed to further advance the technologies that connect audio, vision, and language, which introduces temporal video information and dialogue interactions between a questioner and an answerer. This paper proposes an intuitive mechanism that fuses features and attention in multiple stages in order to well integrate multimodal features, and the results demonstrate its capability in the experiments. Also, we apply several state-of-the-art models in other tasks to the AVSD task, and further analyze their generalization across different tasks.

arXiv commentAccepted for a poster session at the DSTC7 workshop at AAAI 2019

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• Dynamic Memory Networks for Visual and Textual Question Answering
• FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension
• Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7

Cited by in corpus (5)

• Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue
• Multimodal Dialogue State Tracking By QA Approach with Data Augmentation
• TMT: A Transformer-based Modal Translator for Improving Multimodal Sequence Representations in Audio Visual Scene-aware Dialog
• Learning Question-Guided Video Representation for Multi-Turn Video Question Answering
• TCT: A Cross-supervised Learning Method for Multimodal Sequence Representation