DeepStory: Video Story QA by Deep Embedded Memory Networks

Abstract

Question-answering (QA) on video contents is a significant challenge for achieving human-level intelligence as it involves both vision and language in real-world settings. Here we demonstrate the possibility of an AI agent performing video story QA by learning from a large amount of cartoon videos. We develop a video-story learning model, i.e. Deep Embedded Memory Networks (DEMN), to reconstruct stories from a joint scene-dialogue video stream using a latent embedding space of observed data. The video stories are stored in a long-term memory component. For a given question, an LSTM-based attention model uses the long-term memory to recall the best question-story-answer triplet by focusing on specific words containing key information. We trained the DEMN on a novel QA dataset of children's cartoon video series, Pororo. The dataset contains 16,066 scene-dialogue pairs of 20.5-hour videos, 27,328 fine-grained sentences for scene description, and 8,913 story-related QA pairs. Our experimental results show that the DEMN outperforms other QA models. This is mainly due to 1) the reconstruction of video stories in a scene-dialogue combined form that utilize the latent embedding and 2) attention. DEMN also achieved state-of-the-art results on the MovieQA benchmark.

arXiv comment7 pages, accepted for IJCAI 2017

References in corpus (7)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Deep Learning for Answer Sentence Selection
• Coherent Multi-Sentence Video Description with Variable Level of Detail
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• A Dataset for Movie Description
• TGIF: A New Dataset and Benchmark on Animated GIF Description

Cited by in corpus (17)

• Interactive Question Answering Systems: Literature Review
• TVQA: Localized, Compositional Video Question Answering
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System
• A Read-Write Memory Network for Movie Story Understanding
• ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
• Self-supervised pre-training and contrastive representation learning for multiple-choice video QA
• From Visual to Acoustic Question Answering
• TutorialVQA: Question Answering Dataset for Tutorial Videos
• Video Question Generation via Cross-Modal Self-Attention Networks Learning
• Situation and Behavior Understanding by Trope Detection on Films
• Holistic Multi-modal Memory Network for Movie Question Answering
• Adversarial Multimodal Network for Movie Question Answering
• Characterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset
• TrUMAn: Trope Understanding in Movies and Animations
• Adaptive Recurrent Neural Network Based on Mixture Layer