Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment

Abstract

This paper provides an empirical evaluation of recently developed exploration algorithms within the Arcade Learning Environment (ALE). We study the use of different reward bonuses that incentives exploration in reinforcement learning. We do so by fixing the learning algorithm used and focusing only on the impact of the different exploration bonuses in the agent's performance. We use Rainbow, the state-of-the-art algorithm for value-based agents, and focus on some of the bonuses proposed in the last few years. We consider the impact these algorithms have on performance within the popular game Montezuma's Revenge which has gathered a lot of interest from the exploration community, across the the set of seven games identified by Bellemare et al. (2016) as challenging for exploration, and easier games where exploration is not an issue. We find that, in our setting, recently developed bonuses do not provide significantly improved performance on Montezuma's Revenge or hard exploration games. We also find that existing bonus-based methods may negatively impact performance on games in which exploration is not an issue and may even perform worse than $\epsilon$-greedy exploration.

Accepted at the second Exploration in Reinforcement Learning Workshop at the 36th International Conference on Machine Learning, Long Beach, California. The full version arxiv.org/abs/2109.11052 was published as a conference paper at ICLR 2020

References in corpus (2)

• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning

Cited by in corpus (14)

• A survey on intrinsic motivation in reinforcement learning
• Efficient Exploration via State Marginal Matching
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation
• See, Hear, Explore: Curiosity via Audio-Visual Association
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
• Meta-learning curiosity algorithms
• Student-Initiated Action Advising via Advice Novelty
• Self-Imitation Advantage Learning
• Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• On Catastrophic Interference in Atari 2600 Games
• Exploring Unknown States with Action Balance