Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents

Abstract

We investigate the challenge of task planning for multi-task embodied agents in open-world environments. Two main difficulties are identified: 1) executing plans in an open-world environment (e.g., Minecraft) necessitates accurate and multi-step reasoning due to the long-term nature of tasks, and 2) as vanilla planners do not consider how easy the current agent can achieve a given sub-task when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient or even infeasible. To this end, we propose "$\underline{D}$escribe, $\underline{E}$xplain, $\underline{P}$lan and $\underline{S}$elect" ($\text{DEPS}$), an interactive planning approach based on Large Language Models (LLMs). DEPS facilitates better error correction on initial LLM-generated $\text{plan}$ by integrating $\text{description}$ of the plan execution process and providing self-$\text{explanation}$ of feedback when encountering failures during the extended planning phases. Furthermore, it includes a goal $\text{selector}$, which is a trainable module that ranks parallel candidate sub-goals based on the estimated steps of completion, consequently refining the initial plan. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the $\text{ObtainDiamond}$ grand challenge with our approach. The code is released at https://github.com/CraftJarvis/MC-Planner.

NeurIPS 2023

Cited by in corpus (8)

• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• Theory of Mind for Multi-Agent Collaboration via Large Language Models
• On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• CoPAL: Corrective Planning of Robot Actions with Large Language Models
• Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory
• Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments
• Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study