Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
arXiv:1908.08191
Abstract
With increasing information from social media, there are more and more videos available. Therefore, the ability to reason on a video is important and deserves to be discussed. TheDialog System Technology Challenge (DSTC7) (Yoshino et al. 2018) proposed an Audio Visual Scene-aware Dialog (AVSD) task, which contains five modalities including video, dialogue history, summary, and caption, as a scene-aware environment. In this paper, we propose the entropy-enhanced dynamic memory network (DMN) to effectively model video modality. The attention-based GRU in the proposed model can improve the model's ability to comprehend and memorize sequential information. The entropy mechanism can control the attention distribution higher, so each to-be-answered question can focus more specifically on a small set of video segments. After the entropy-enhanced DMN secures the video context, we apply an attention model that in-corporates summary and caption to generate an accurate answer given the question about the video. In the official evaluation, our system can achieve improved performance against the released baseline model for both subjective and objective evaluation metrics.
arXiv commentDSTC7 collocated with AAAI2019
References in corpus (7)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Neural Machine Translation by Jointly Learning to Align and Translate
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- VQA: Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
Cited by in corpus (6)
- OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
- Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue
- Multimodal Dialogue State Tracking By QA Approach with Data Augmentation
- HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue
- HSCJN: A Holistic Semantic Constraint Joint Network for Diverse Response Generation
- Learning Question-Guided Video Representation for Multi-Turn Video Question Answering