Pushing it out of the Way: Interactive Visual Navigation

Abstract

We have observed significant progress in visual navigation for embodied agents. A common assumption in studying visual navigation is that the environments are static; this is a limiting assumption. Intelligent navigation may involve interacting with the environment beyond just moving forward/backward and turning left/right. Sometimes, the best way to navigate is to push something out of the way. In this paper, we study the problem of interactive navigation where agents learn to change the environment to navigate more efficiently to their goals. To this end, we introduce the Neural Interaction Engine (NIE) to explicitly predict the change in the environment caused by the agent's actions. By modeling the changes while planning, we find that agents exhibit significant improvements in their navigational capabilities. More specifically, we consider two downstream tasks in the physics-enabled, visually rich, AI2-THOR environment: (1) reaching a target while the path to the target is blocked (2) moving an object to a target location by pushing it. For both tasks, agents equipped with an NIE significantly outperform agents without the understanding of the effect of the actions indicating the benefits of our approach.

arXiv comment14 pages, 13 figures, CVPR 2021, https://prior.allenai.org/projects/interactive-visual-navigation, https://youtu.be/GvTI5XCMvPw

References in corpus (14)

• On Evaluation of Embodied Navigation Agents
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Learning to Explore using Active Neural SLAM
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
• ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
• Unsupervised Learning of Object Keypoints for Perception and Control
• Rearrangement: A Challenge for Embodied AI
• AllenAct: A Framework for Embodied AI Research
• Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
• Neural Predictive Belief Representations
• Visual Grounding of Learned Physical Models
• Physically Embedded Planning Problems: New Challenges for Reinforcement Learning
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban