MarioQA: Answering Questions by Watching Gameplay Videos
arXiv:1612.01669
Abstract
We present a framework to analyze various aspects of models for video question answering (VideoQA) using customizable synthetic datasets, which are constructed automatically from gameplay videos. Our work is motivated by the fact that existing models are often tested only on datasets that require excessively high-level reasoning or mostly contain instances accessible through single frame inferences. Hence, it is difficult to measure capacity and flexibility of trained models, and existing techniques often rely on ad-hoc implementations of deep neural networks without clear insight into datasets and models. We are particularly interested in understanding temporal relationships between video events to solve VideoQA problems; this is because reasoning temporal dependency is one of the most distinct components in videos from images. To address this objective, we automatically generate a customized synthetic VideoQA dataset using {\em Super Mario Bros.} gameplay videos so that it contains events with different levels of reasoning complexity. Using the dataset, we show that properly constructed datasets with events in various complexity levels are critical to learn effective models and improve overall performance.
References in corpus (18)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Two-Stream Convolutional Networks for Action Recognition in Videos
- Going Deeper with Convolutions
- VQA: Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Simple Baseline for Visual Question Answering
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- Stacked Attention Networks for Image Question Answering
- Learning Spatiotemporal Features with 3D Convolutional Networks
- Learning to Compose Neural Networks for Question Answering
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
- A Dataset for Movie Description
- Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- Uncovering Temporal Context for Video Question and Answering
- Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
- Face Attribute Prediction Using Off-the-Shelf CNN Features