Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions
arXiv:1809.03707
Abstract
In-depth scene descriptions and question answering tasks have greatly increased the scope of today's definition of scene understanding. While such tasks are in principle open ended, current formulations primarily focus on describing only the current state of the scenes under consideration. In contrast, in this paper, we focus on the future states of the scenes which are also conditioned on actions. We posit this as a question answering task, where an answer has to be given about a future scene state, given observations of the current scene, and a question that includes a hypothetical action. Our solution is a hybrid model which integrates a physics engine into a question answering architecture in order to anticipate future scene states resulting from object-object interactions caused by an action. We demonstrate first results on this challenging new problem and compare to baselines, where we outperform fully data-driven end-to-end learning approaches.
arXiv commentPaper: 18 pages, 5 figures, 5 tables. Supplementary material: 3 pages, 1 figure, 1 table. To be published in VLEASE ECCV 2018 workshop
References in corpus (9)
- VQA: Visual Question Answering
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Video (language) modeling: a baseline for generative models of natural videos
- Building Generalizable Agents with a Realistic and Rich 3D Environment
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction
- DeepMind Lab
- DVQA: Understanding Data Visualizations via Question Answering
Cited by in corpus (5)
- A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories
- AVoE: A Synthetic 3D Dataset on Understanding Violation of Expectation for Artificial Cognition
- Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning
- Understanding in Artificial Intelligence
- Non-monotonic Logical Reasoning Guiding Deep Learning for Explainable Visual Question Answering