TVQA: Localized, Compositional Video Question Answering

Abstract

Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks. However, due to data limitations, there has been much less work on video-based QA. In this paper, we present TVQA, a large-scale video QA dataset based on 6 popular TV shows. TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video. Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts. We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task. The dataset is publicly available at http://tvqa.cs.unc.edu.

arXiv commentEMNLP 2018 (13 pages; Data and Leaderboard at: http://tvqa.cs.unc.edu). Updated with test-public results

References in corpus (8)

• Teaching Machines to Read and Comprehend
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• MAttNet: Modular Attention Network for Referring Expression Comprehension
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• Modeling Context in Referring Expressions
• A Dataset for Movie Description
• DeepStory: Video Story QA by Deep Embedded Memory Networks

Cited by in corpus (17)

• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Interactive Question Answering Systems: Literature Review
• iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering
• Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
• MCQA: Multimodal Co-attention Based Network for Question Answering
• VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
• Support-Set Based Cross-Supervision for Video Grounding
• BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining
• Self-supervised pre-training and contrastive representation learning for multiple-choice video QA
• Winning the ICCV'2021 VALUE Challenge: Task-aware Ensemble and Transfer Learning with Visual Concepts
• TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
• Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering
• ActBERT: Learning Global-Local Video-Text Representations
• Characterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset
• Analysis on Image Set Visual Question Answering