Reflexion: Language Agents with Verbal Reinforcement Learning

Abstract

Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.

arXiv commentv4 contains a few additional experiments

Cited by in corpus (27)

• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
• Materials science in the era of large language models: a perspective
• A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
• Evaluation of large language models for discovery of gene set function
• Synthetic Data Generation Using Large Language Models: Advances in Text and Code
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• REFINER: Reasoning Feedback on Intermediate Representations
• Domain-specific ChatBots for Science using Embeddings
• Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency
• Reasoning Beyond Limits: Advances and Open Problems for LLMs
• Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
• Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments
• Automating Traffic Model Enhancement with AI Research Agent
• Emergent Language: A Survey and Taxonomy
• Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
• Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models
• Flows: Building Blocks of Reasoning and Collaborating AI
• DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries
• TickIt: Leveraging Large Language Models for Automated Ticket Escalation
• Camera Control at the Edge with Language Models for Scene Understanding
• SituationalLLM: Proactive language models with scene awareness for dynamic, contextual task guidance
• Thought Flow Nets: From Single Predictions to Trains of Model Thought
• Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation
• Real-Time Procedural Learning From Experience for AI Agents
• Prime the search: Using large language models for guiding geometric task and motion planning by warm-starting tree search