DramaQA: Character-Centered Video Story Understanding with Hierarchical QA

Abstract

Despite recent progress on computer vision and natural language processing, developing a machine that can understand video story is still hard to achieve due to the intrinsic difficulty of video story. Moreover, researches on how to evaluate the degree of video understanding based on human cognitive process have not progressed as yet. In this paper, we propose a novel video question answering (Video QA) task, DramaQA, for a comprehensive understanding of the video story. The DramaQA focuses on two perspectives: 1) Hierarchical QAs as an evaluation metric based on the cognitive developmental stages of human intelligence. 2) Character-centered video annotations to model local coherence of the story. Our dataset is built upon the TV drama "Another Miss Oh" and it contains 17,983 QA pairs from 23,928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217,308 annotated images with rich character-centered annotations, including visual bounding boxes, behaviors and emotions of main characters, and coreference resolved scripts. Additionally, we suggest Multi-level Context Matching model which hierarchically understands character-centered representations of video to answer questions. We release our dataset and model publicly for research purposes, and we expect our work to provide a new perspective on video story understanding research.

arXiv comment15 pages, 11 figures, accepted to AAAI 2021

References in corpus (15)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Teaching Machines to Read and Comprehend
• Bidirectional Attention Flow for Machine Comprehension
• YouTube-8M: A Large-Scale Video Classification Benchmark
• End-To-End Memory Networks
• Narrative Planning: Balancing Plot and Character
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• NewsQA: A Machine Comprehension Dataset
• Computational Narrative Intelligence: A Human-Centered Goal for Artificial Intelligence
• TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning
• Multimodal Dual Attention Memory for Video Story Question Answering
• Constructing Hierarchical Q&A Datasets for Video Story Understanding

Cited by in corpus (4)

• MERLOT: Multimodal Neural Script Knowledge Models
• BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining
• Mounting Video Metadata on Transformer-based Language Model for Open-ended Video Question Answering
• Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering