Inner Monologue: Embodied Reasoning through Planning with Language Models

Abstract

Recent works have shown how the reasoning capabilities of Large Language Models (LLMs) can be applied to domains beyond natural language processing, such as planning and interaction for robots. These embodied problems require an agent to understand many semantic aspects of the world: the repertoire of skills available, how these skills influence the world, and how changes to the world map back to the language. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them - answers that change over time in response to the agent's own choices. In this work, we investigate to what extent LLMs used in such embodied contexts can reason over sources of feedback provided through natural language, without any additional training. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios. We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction. We find that closed-loop language feedback significantly improves high-level instruction completion on three domains, including simulated and real table top rearrangement tasks and long-horizon mobile manipulation tasks in a kitchen environment in the real world.

arXiv commentProject website: https://innermonologue.github.io

Cited by in corpus (19)

• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• Large Language Models and the Reverse Turing Test
• Harms from Increasingly Agentic Algorithmic Systems
• ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
• Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation
• Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worlds
• On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)
• LLM-BT: Performing Robotic Adaptive Tasks based on Large Language Models and Behavior Trees
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models
• CoPAL: Corrective Planning of Robot Actions with Large Language Models
• Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory
• Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?
• Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes
• LEMMA: Learning Language-Conditioned Multi-Robot Manipulation
• TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models
• AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agents
• LLM-GROP: Visually Grounded Robot Task and Motion Planning with Large Language Models
• Remember what you did so you know what to do next