On Evaluation of Embodied Navigation Agents

Abstract

Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence. The past two years have seen a surge of creative work on navigation. This creative output has produced a plethora of sometimes incompatible task definitions and evaluation protocols. To coordinate ongoing and future research in this area, we have convened a working group to study empirical methodology in navigation research. The present document summarizes the consensus recommendations of this working group. We discuss different problem statements and the role of generalization, present evaluation measures, and provide standard scenarios that can be used for benchmarking.

arXiv commentReport of a working group on empirical methodology in navigation research. Authors are listed in alphabetical order

References in corpus (12)

• Learning to Navigate in Complex Environments
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• AI2-THOR: An Interactive 3D Environment for Visual AI
• CARLA: An Open Urban Driving Simulator
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• Driving Policy Transfer via Modularity and Abstraction
• Control of Memory, Active Perception, and Action in Minecraft
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• Neural Map: Structured Memory for Deep Reinforcement Learning
• HoME: a Household Multimodal Environment
• Unifying Map and Landmark Based Representations for Visual Navigation
• Semi-parametric Topological Memory for Navigation

Cited by in corpus (132)

• The Replica Dataset: A Digital Replica of Indoor Spaces
• Object Goal Navigation using Goal-Oriented Semantic Exploration
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Rearrangement: A Challenge for Embodied AI
• L3MVN: Leveraging Large Language Models for Visual Target Navigation
• Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation
• Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
• Talk2Nav: Long-Range Vision-and-Language Navigation with Dual Attention and Spatial Memory
• Chasing Ghosts: Instruction Following as Bayesian State Tracking
• Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
• AllenAct: A Framework for Embodied AI Research
• iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
• Benchmarking Classic and Learned Navigation in Complex 3D Environments
• iPlanner: Imperative Path Planning
• Embodied Visual Navigation with Automatic Curriculum Learning in Real Environments
• A Recurrent Vision-and-Language BERT for Navigation
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Learning Exploration Policies for Navigation
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
• Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
• Learning hierarchical relationships for object-goal navigation
• Learning to Set Waypoints for Audio-Visual Navigation
• Frontier Semantic Exploration for Visual Target Navigation
• VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects
• A Survey of Embodied AI: From Simulators to Research Tasks
• Habitat 2.0: Training Home Assistants to Rearrange their Habitat
• Semantic Visual Navigation by Watching YouTube Videos
• LanguageRefer: Spatial-Language Model for 3D Visual Grounding
• Path Planning using Neural A* Search
• Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships
• Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments
• Sim-to-Real Transfer for Vision-and-Language Navigation
• Learning to Seek: Autonomous Source Seeking with Deep Reinforcement Learning Onboard a Nano Drone Microcontroller
• Language and Visual Entity Relationship Graph for Agent Navigation
• Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters
• Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulation
• Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
• Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agents
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Collaborative Visual Navigation
• Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling
• To Learn or Not to Learn: Analyzing the Role of Learning for Navigation in Virtual Environments
• Auxiliary Tasks and Exploration Enable ObjectNav
• Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
• Learning to Navigate in Indoor Environments: from Memorizing to Reasoning
• FILM: Following Instructions in Language with Modular Methods
• Unsupervised Visual Odometry and Action Integration for PointGoal Navigation in Indoor Environment
• The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation
• Deep Learning for Embodied Vision Navigation: A Survey
• What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning
• Learning Navigation Subroutines from Egocentric Videos
• Topological Planning with Transformers for Vision-and-Language Navigation
• Learning to Navigate Using Mid-Level Visual Priors
• Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
• Multi-View Learning for Vision-and-Language Navigation
• Learning Semantic-Agnostic and Spatial-Aware Representation for Generalizable Visual-Audio Navigation
• Open Scene Graphs for Open-World Object-Goal Navigation
• VisualEchoes: Spatial Image Representation Learning through Echolocation
• Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
• Using RGB Image as Visual Input for Mapless Robot Navigation
• Multi-modal Discriminative Model for Vision-and-Language Navigation
• Learning to Visually Navigate in Photorealistic Environments Without any Supervision
• Implicit Obstacle Map-driven Indoor Navigation Model for Robust Obstacle Avoidance
• Leveraging Large Language Model-based Room-Object Relationships Knowledge for Enhancing Multimodal-Input Object Goal Navigation
• MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigation
• Where are the Keys? -- Learning Object-Centric Navigation Policies on Semantic Maps with Graph Convolutional Networks
• No RL, No Simulation: Learning to Navigate without Navigating
• VALAN: Vision and Language Agent Navigation
• Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation
• Bayesian Relational Memory for Semantic Visual Navigation
• Object-and-Action Aware Model for Visual Language Navigation
• SoundSpaces: Audio-Visual Navigation in 3D Environments
• An A* Curriculum Approach to Reinforcement Learning for RGBD Indoor Robot Navigation
• Unsupervised Active Visual Search with Monte Carlo planning under Uncertain Detections
• Hierarchical Path-planning from Speech Instructions with Spatial Concept-based Topometric Semantic Mapping
• Unsupervised Domain Adaptation for Visual Navigation
• Auxiliary Tasks Speed Up Learning PointGoal Navigation
• Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
• Transferable Representation Learning in Vision-and-Language Navigation
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration
• Soft Expert Reward Learning for Vision-and-Language Navigation
• Learning Object-conditioned Exploration using Distributed Soft Actor Critic
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments
• Bootstrapping Cognitive Agents with a Large Language Model
• Learning to Map for Active Semantic Goal Navigation
• Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information
• SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments
• SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation
• An Inductive Bias for Distances: Neural Nets that Respect the Triangle Inequality
• I2EDL: Interactive Instruction Error Detection and Localization
• Spot the Difference: A Novel Task for Embodied Agents in Changing Environments
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• RoCUS: Robot Controller Understanding via Sampling
• Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rule
• MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Cross-Lingual Vision-Language Navigation
• Deep Visual MPC-Policy Learning for Navigation
• Rethinking the Spatial Route Prior in Vision-and-Language Navigation
• NeoNav: Improving the Generalization of Visual Navigation via Generating Next Expected Observations
• The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation
• Multi-Agent Embodied Visual Semantic Navigation with Scene Prior Knowledge
• The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigation
• Analyzing Visual Representations in Embodied Navigation Tasks
• Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation
• Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning
• A Few Shot Adaptation of Visual Navigation Skills to New Observations using Meta-Learning
• Towards Target-Driven Visual Navigation in Indoor Scenes via Generative Imitation Learning
• Semantic Audio-Visual Navigation
• Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms
• PanGEA: The Panoramic Graph Environment Annotation Toolkit
• Less is More: Generating Grounded Navigation Instructions from Landmarks
• Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
• Pushing it out of the Way: Interactive Visual Navigation
• Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments
• Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation
• Multimodal Attention Networks for Low-Level Vision-and-Language Navigation
• Realistic PointGoal Navigation via Auxiliary Losses and Information Bottleneck
• Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph
• Maximizing BCI Human Feedback using Active Learning
• Natural Language for Human-Robot Collaboration: Problems Beyond Language Grounding
• Building Intelligent Autonomous Navigation Agents
• Learning Synthetic to Real Transfer for Localization and Navigational Tasks
• Learning Composable Behavior Embeddings for Long-horizon Visual Navigation
• Towards real-world navigation with deep differentiable planners