On Bonus-Based Exploration Methods in the Arcade Learning Environment

Abstract

Research on exploration in reinforcement learning, as applied to Atari 2600 game-playing, has emphasized tackling difficult exploration problems such as Montezuma's Revenge (Bellemare et al., 2016). Recently, bonus-based exploration methods, which explore by augmenting the environment reward, have reached above-human average performance on such domains. In this paper we reassess popular bonus-based exploration methods within a common evaluation framework. We combine Rainbow (Hessel et al., 2018) with different exploration bonuses and evaluate its performance on Montezuma's Revenge, Bellemare et al.'s set of hard of exploration games with sparse rewards, and the whole Atari 2600 suite. We find that while exploration bonuses lead to higher score on Montezuma's Revenge they do not provide meaningful gains over the simpler $\epsilon$-greedy scheme. In fact, we find that methods that perform best on that game often underperform $\epsilon$-greedy on easy exploration Atari 2600 games. We find that our conclusions remain valid even when hyperparameters are tuned for these easy-exploration games. Finally, we find that none of the methods surveyed benefit from additional training samples (1 billion frames, versus Rainbow's 200 million) on Bellemare et al.'s hard exploration games. Our results suggest that recent gains in Montezuma's Revenge may be better attributed to architecture change, rather than better exploration schemes; and that the real pace of progress in exploration research for Atari 2600 games may have been obfuscated by good results on a single domain.

Full version of arXiv:1908.02388

References in corpus (9)

• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Large-Scale Study of Curiosity-Driven Learning
• Deep Reinforcement Learning that Matters
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Go-Explore: a New Approach for Hard-Exploration Problems
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Behaviour Suite for Reinforcement Learning

Cited by in corpus (10)

• Mastering Atari with Discrete World Models
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Evaluating Agents without Rewards
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• Novelty Search in Representational Space for Sample Efficient Exploration
• Principled Exploration via Optimistic Bootstrapping and Backward Induction
• Temporally-Extended ε-Greedy Exploration
• State-Aware Variational Thompson Sampling for Deep Q-Networks
• Perturbation-based exploration methods in deep reinforcement learning
• Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning