Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field

Abstract

Consistent and reproducible evaluation of Deep Reinforcement Learning (DRL) is not straightforward. In the Arcade Learning Environment (ALE), small changes in environment parameters such as stochasticity or the maximum allowed play time can lead to very different performance. In this work, we discuss the difficulties of comparing different agents trained on ALE. In order to take a step further towards reproducible and comparable DRL, we introduce SABER, a Standardized Atari BEnchmark for general Reinforcement learning algorithms. Our methodology extends previous recommendations and contains a complete set of environment parameters as well as train and test procedures. We then use SABER to evaluate the current state of the art, Rainbow. Furthermore, we introduce a human world records baseline, and argue that previous claims of expert or superhuman performance of DRL might not be accurate. Finally, we propose Rainbow-IQN by extending Rainbow with Implicit Quantile Networks (IQN) leading to new state-of-the-art performance. Source code is available for reproducibility.

Paper currently in review

References in corpus (11)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Prioritized Experience Replay
• Noisy Networks for Exploration
• Distributed Prioritized Experience Replay
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Deep Reinforcement Learning that Matters
• Count-Based Exploration with Neural Density Models
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Go-Explore: a New Approach for Hard-Exploration Problems
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Observe and Look Further: Achieving Consistent Performance on Atari

Cited by in corpus (9)

• Agent57: Outperforming the Atari Human Benchmark
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Mastering Atari with Discrete World Models
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning
• An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi
• More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences