Enhancing Reinforcement Learning Through Guided Search

Abstract

With the aim of improving performance in Markov Decision Problem in an Off-Policy setting, we suggest taking inspiration from what is done in Offline Reinforcement Learning (RL). In Offline RL, it is a common practice during policy learning to maintain proximity to a reference policy to mitigate uncertainty, reduce potential policy errors, and help improve performance. We find ourselves in a different setting, yet it raises questions about whether a similar concept can be applied to enhance performance ie, whether it is possible to find a guiding policy capable of contributing to performance improvement, and how to incorporate it into our RL agent. Our attention is particularly focused on algorithms based on Monte Carlo Tree Search (MCTS) as a guide.MCTS renowned for its state-of-the-art capabilities across various domains, catches our interest due to its ability to converge to equilibrium in single-player and two-player contexts. By harnessing the power of MCTS as a guide for our RL agent, we observed a significant performance improvement, surpassing the outcomes achieved by utilizing each method in isolation. Our experiments were carried out on the Atari 100k benchmark.

Accepted Paper at ECAI 2024; Extended Version

References in corpus (18)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Prioritized Experience Replay
• Soft Actor-Critic Algorithms and Applications
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• Off-Policy Deep Reinforcement Learning without Exploration
• Behavior Regularized Offline Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Mastering Diverse Domains through World Models
• Mastering Atari Games with Limited Data
• Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning
• Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
• Batch Monte Carlo Tree Search