Multimodal Dialogue State Tracking By QA Approach with Data Augmentation

Abstract

Recently, a more challenging state tracking task, Audio-Video Scene-Aware Dialogue (AVSD), is catching an increasing amount of attention among researchers. Different from purely text-based dialogue state tracking, the dialogue in AVSD contains a sequence of question-answer pairs about a video and the final answer to the given question requires additional understanding of the video. This paper interprets the AVSD task from an open-domain Question Answering (QA) point of view and proposes a multimodal open-domain QA system to deal with the problem. The proposed QA system uses common encoder-decoder framework with multimodal fusion and attention. Teacher forcing is applied to train a natural language generator. We also propose a new data augmentation approach specifically under QA assumption. Our experiments show that our model and techniques bring significant improvements over the baseline model on the DSTC7-AVSD dataset and demonstrate the potentials of our data augmentation techniques.

AAAI DSTC8 Workshop

References in corpus (8)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• VQA: Visual Question Answering
• Machine Comprehension Using Match-LSTM and Answer Pointer
• Image Captioning with Semantic Attention
• CIDEr: Consensus-based Image Description Evaluation
• Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
• Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modeling

Cited by in corpus (2)

• Data Augmentation Approaches in Natural Language Processing: A Survey
• Slot Self-Attentive Dialogue State Tracking