Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation

Abstract

We present LGX (Language-guided Exploration), a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON), where an embodied agent navigates to a uniquely described target object in a previously unseen environment. Our approach makes use of Large Language Models (LLMs) for this task by leveraging the LLM's commonsense reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach via \textit{real-world} experiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.

10 pages

References in corpus (7)

• RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Term Online Operation
• Large Language Models Are Human-Level Prompt Engineers
• Inner Monologue: Embodied Reasoning through Planning with Language Models
• ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
• Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection
• Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worlds
• Large Language Models as Commonsense Knowledge for Large-Scale Task Planning

Cited by in corpus (7)

• Open Scene Graphs for Open-World Object-Goal Navigation
• ROSGPT_Vision: Commanding Robots Using Only Language Models' Prompts
• When control meets large language models: From words to dynamics
• TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goals
• Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning
• HomeEmergency -- Using Audio to Find and Respond to Emergencies in the Home
• Balancing Performance and Efficiency in Zero-shot Robotic Navigation