Motion-Appearance Co-Memory Networks for Video Question Answering
arXiv:1803.10906
Abstract
Video Question Answering (QA) is an important task in understanding video temporal structure. We observe that there are three unique attributes of video QA compared with image QA: (1) it deals with long sequences of images containing richer information not only in quantity but also in variety; (2) motion and appearance information are usually correlated with each other and able to provide useful attention cues to the other; (3) different questions require different number of frames to infer the answer. Based these observations, we propose a motion-appearance comemory network for video QA. Our networks are built on concepts from Dynamic Memory Network (DMN) and introduces new mechanisms for video QA. Specifically, there are three salient aspects: (1) a co-memory attention mechanism that utilizes cues from both motion and appearance to generate attention; (2) a temporal conv-deconv network to generate multi-level contextual facts; (3) a dynamic fact ensemble method to construct temporal representation dynamically for different questions. We evaluate our method on TGIF-QA dataset, and the results outperform state-of-the-art significantly on all four tasks of TGIF-QA.
arXiv commentCVPR 2018
References in corpus (28)
- Adam: A Method for Stochastic Optimization
- Deep Residual Learning for Image Recognition
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- Multimodal Residual Learning for Visual QA
- Stacked Attention Networks for Image Question Answering
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
- Learning Spatiotemporal Features with 3D Convolutional Networks
- CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016
- A Focused Dynamic Attention Model for Visual Question Answering
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
- TALL: Temporal Activity Localization via Language Query
- TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
- Cascaded Boundary Regression for Temporal Action Detection
- Visual7W: Grounded Question Answering in Images
- TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
- DeepStory: Video Story QA by Deep Embedded Memory Networks
- Visual Semantic Planning using Deep Successor Representations
- Structured Attentions for Visual Question Answering
- End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering
- A Read-Write Memory Network for Movie Story Understanding