Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory

Abstract

The captivating realm of Minecraft has attracted substantial research interest in recent years, serving as a rich platform for developing intelligent agents capable of functioning in open-world environments. However, the current research landscape predominantly focuses on specific objectives, such as the popular "ObtainDiamond" task, and has not yet shown effective generalization to a broader spectrum of tasks. Furthermore, the current leading success rate for the "ObtainDiamond" task stands at around 20%, highlighting the limitations of Reinforcement Learning (RL) based controllers used in existing methods. To tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel framework integrates Large Language Models (LLMs) with text-based knowledge and memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These agents, equipped with the logic and common sense capabilities of LLMs, can skillfully navigate complex, sparse-reward environments with text-based interactions. We develop a set of structured actions and leverage LLMs to generate action plans for the agents to execute. The resulting LLM-based agent markedly surpasses previous methods, achieving a remarkable improvement of +47.5% in success rate on the "ObtainDiamond" task, demonstrating superior robustness compared to traditional RL-based controllers. Notably, our agent is the first to procure all items in the Minecraft Overworld technology tree, demonstrating its extensive capabilities. GITM does not need any GPU for training, but a single CPU node with 32 CPU cores is enough. This research shows the potential of LLMs in developing capable agents for handling long-horizon, complex tasks and adapting to uncertainties in open-world environments. See the project website at https://github.com/OpenGVLab/GITM.

References in corpus (19)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Toolformer: Language Models Can Teach Themselves to Use Tools
• PaLM-E: An Embodied Multimodal Language Model
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
• Inner Monologue: Embodied Reasoning through Planning with Language Models
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
• Mastering Diverse Domains through World Models
• MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
• Open-Ended Learning Leads to Generally Capable Agents
• Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
• Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
• Skill Reinforcement Learning and Planning for Open-World Long-Horizon Tasks
• Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft
• Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning
• Scaling Imitation Learning in Minecraft
• MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learned
• JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning
• Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition

Cited by in corpus (2)

• Reward Guidance for Reinforcement Learning Tasks Based on Large Language Models: The LMGT Framework
• Bootstrapping Cognitive Agents with a Large Language Model