Location-aware Graph Convolutional Networks for Video Question Answering
arXiv:2008.09105
Abstract
We addressed the challenging task of video question answering, which requires machines to answer questions about videos in a natural language form. Previous state-of-the-art methods attempt to apply spatio-temporal attention mechanism on video frame features without explicitly modeling the location and relations among object interaction occurred in videos. However, the relations between object interaction and their location information are very critical for both action recognition and question reasoning. In this work, we propose to represent the contents in the video as a location-aware graph by incorporating the location information of an object into the graph construction. Here, each node is associated with an object represented by its appearance and location features. Based on the constructed graph, we propose to use graph convolution to infer both the category and temporal locations of an action. As the graph is built on objects, our method is able to focus on the foreground action contents for better video question answering. Lastly, we leverage an attention mechanism to combine the output of graph convolution and encoded question features for final answer reasoning. Extensive experiments demonstrate the effectiveness of the proposed methods. Specifically, our method significantly outperforms state-of-the-art methods on TGIF-QA, Youtube2Text-QA, and MSVD-QA datasets. Code and pre-trained models are publicly available at: https://github.com/SunDoge/L-GCN
References in corpus (11)
- Dynamic Memory Networks for Visual and Textual Question Answering
- Learning Conditioned Graph Structures for Interpretable Visual Question Answering
- NAT: Neural Architecture Transformer for Accurate and Compact Architectures
- CLEVRER: CoLlision Events for Video REpresentation and Reasoning
- TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
- Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
- Multi-marginal Wasserstein GAN
- Adversarial Learning with Local Coordinate Coding
- VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
- MUREL: Multimodal Relational Reasoning for Visual Question Answering
- Self-supervised Moving Vehicle Tracking with Stereo Sound