Learning Question-Guided Video Representation for Multi-Turn Video Question Answering
arXiv:1907.13280
Abstract
Understanding and conversing about dynamic scenes is one of the key capabilities of AI agents that navigate the environment and convey useful information to humans. Video question answering is a specific scenario of such AI-human interaction where an agent generates a natural language response to a question regarding the video of a dynamic scene. Incorporating features from multiple modalities, which often provide supplementary information, is one of the challenging aspects of video question answering. Furthermore, a question often concerns only a small segment of the video, hence encoding the entire video sequence using a recurrent neural network is not computationally efficient. Our proposed question-guided video representation module efficiently generates the token-level video summary guided by each word in the question. The learned representations are then fused with the question to generate the answer. Through empirical evaluation on the Audio Visual Scene-aware Dialog (AVSD) dataset, our proposed models in single-turn and multi-turn question answering achieve state-of-the-art performance on several automatic natural language generation evaluation metrics.
Accepted at SIGDIAL 2019
References in corpus (15)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- On the difficulty of training Recurrent Neural Networks
- The Kinetics Human Action Video Dataset
- Bidirectional Attention Flow for Machine Comprehension
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- End-To-End Memory Networks
- Exploring Models and Data for Image Question Answering
- Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
- Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
- Video Question Answering via Attribute-Augmented Attention Network Learning
- Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
- Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modeling
- Context, Attention and Audio Feature Explorations for Audio Visual Scene-Aware Dialog
- End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features