VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering

Abstract

Embodied Question Answering (EQA) is a recently proposed task, where an agent is placed in a rich 3D environment and must act based solely on its egocentric input to answer a given question. The desired outcome is that the agent learns to combine capabilities such as scene understanding, navigation and language understanding in order to perform complex reasoning in the visual world. However, initial advancements combining standard vision and language methods with imitation and reinforcement learning algorithms have shown EQA might be too complex and challenging for these techniques. In order to investigate the feasibility of EQA-type tasks, we build the VideoNavQA dataset that contains pairs of questions and videos generated in the House3D environment. The goal of this dataset is to assess question-answering performance from nearly-ideal navigation paths, while considering a much more complete variety of questions than current instantiations of the EQA task. We investigate several models, adapted from popular VQA methods, on this new benchmark. This establishes an initial understanding of how well VQA-style methods can perform within this novel EQA paradigm.

arXiv commentTo appear at BMVC 2019. 15 pages, 5 figures

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Exploring Models and Data for Image Question Answering
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Compositional Attention Networks for Machine Reasoning
• Benchmarking Classic and Learned Navigation in Complex 3D Environments
• Blindfold Baselines for Embodied QA
• Systematic Generalization: What Is Required and Can It Be Learned?
• From FiLM to Video: Multi-turn Question Answering with Multi-modal Context

Cited by in corpus (3)

• Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
• Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation
• Probing Emergent Semantics in Predictive Agents via Question Answering