TVQA+: Spatio-Temporal Grounding for Video Question Answering
arXiv:1904.11574
Abstract
We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos. We first augment the TVQA dataset with 310.8K bounding boxes, linking depicted objects to visual concepts in questions and answers. We name this augmented version as TVQA+. We then propose Spatio-Temporal Answerer with Grounded Evidence (STAGE), a unified framework that grounds evidence in both spatial and temporal domains to answer questions about videos. Comprehensive experiments and analyses demonstrate the effectiveness of our framework and how the rich annotations in our TVQA+ dataset can contribute to the question answering task. Moreover, by performing this joint task, our model is able to produce insightful and interpretable spatio-temporal attention visualizations. Dataset and code are publicly available at: http: //tvqa.cs.unc.edu, https://github.com/jayleicn/TVQAplus
arXiv commentACL 2020 camera-ready (15 pages)
References in corpus (10)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Bidirectional Attention Flow for Machine Comprehension
- VQA: Visual Question Answering
- Layer Normalization
- Know What You Don't Know: Unanswerable Questions for SQuAD
- BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
- Attention Correctness in Neural Image Captioning
- Modeling Context in Referring Expressions
- Interpretable Counting for Visual Question Answering
- TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Cited by in corpus (15)
- MERLOT: Multimodal Neural Script Knowledge Models
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
- Video Understanding as Machine Translation
- iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering
- LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
- MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
- DramaQA: Character-Centered Video Story Understanding with Hierarchical QA
- VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
- VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
- Self-supervised pre-training and contrastive representation learning for multiple-choice video QA
- TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
- Alleviating the Burden of Labeling: Sentence Generation by Attention Branch Encoder-Decoder Network
- On the Learning Dynamics of Attention Networks
- Modality Shifting Attention Network for Multi-modal Video Question Answering
- A Multimodal Target-Source Classifier with Attention Branches to Understand Ambiguous Instructions for Fetching Daily Objects