Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods

Abstract

With extensive pre-trained knowledge and high-level general capabilities, large language models (LLMs) emerge as a promising avenue to augment reinforcement learning (RL) in aspects such as multi-task learning, sample efficiency, and high-level task planning. In this survey, we provide a comprehensive review of the existing literature in LLM-enhanced RL and summarize its characteristics compared to conventional RL methods, aiming to clarify the research scope and directions for future studies. Utilizing the classical agent-environment interaction paradigm, we propose a structured taxonomy to systematically categorize LLMs' functionalities in RL, including four roles: information processor, reward designer, decision-maker, and generator. For each role, we summarize the methodologies, analyze the specific RL challenges that are mitigated, and provide insights into future directions. Lastly, a comparative analysis of each role, potential applications, prospective opportunities, and challenges of the LLM-enhanced RL are discussed. By proposing this taxonomy, we aim to provide a framework for researchers to effectively leverage LLMs in the RL field, potentially accelerating RL applications in complex applications such as robotics, autonomous driving, and energy systems.

22 pages (including bibliography), 6 figures

References in corpus (8)

• A Survey on Large Language Model based Autonomous Agents
• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• What learning algorithm is in-context learning? Investigations with linear models
• Goal Misgeneralization in Deep Reinforcement Learning
• SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference
• Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents
• Towards Optimizing Human-Centric Objectives in AI-Assisted Decision-Making With Offline Reinforcement Learning

Cited by in corpus (5)

• Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
• A Review of Safe Reinforcement Learning Methods for Modern Power Systems
• Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap
• MCP-Enabled LLM for Meta-optics Inverse Design: Leveraging Differentiable Solver without LLM Expertise
• Endogenous Regime Switching Driven by Scalar-Irreducible Learning Dynamics