Towards robust and domain agnostic reinforcement learning competitions

Abstract

Reinforcement learning competitions have formed the basis for standard research benchmarks, galvanized advances in the state-of-the-art, and shaped the direction of the field. Despite this, a majority of challenges suffer from the same fundamental problems: participant solutions to the posed challenge are usually domain-specific, biased to maximally exploit compute resources, and not guaranteed to be reproducible. In this paper, we present a new framework of competition design that promotes the development of algorithms that overcome these barriers. We propose four central mechanisms for achieving this end: submission retraining, domain randomization, desemantization through domain obfuscation, and the limitation of competition compute and environment-sample budget. To demonstrate the efficacy of this design, we proposed, organized, and ran the MineRL 2020 Competition on Sample-Efficient Reinforcement Learning. In this work, we describe the organizational outcomes of the competition and show that the resulting participant submissions are reproducible, non-specific to the competition environment, and sample/resource efficient, despite the difficult competition task.

arXiv comment20 pages, several figures, published PMLR

References in corpus (23)

• Dueling Network Architectures for Deep Reinforcement Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• Conservative Q-Learning for Offline Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Deep Q-learning from Demonstrations
• Control of Memory, Active Perception, and Action in Minecraft
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Investigating Human Priors for Playing Video Games
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Decoupling Representation Learning from Reinforcement Learning
• The Multi-Agent Reinforcement Learning in MalmÖ (MARLÖ) Competition
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors
• Skynet: A Top Deep RL Agent in the Inaugural Pommerman Team Competition
• Flatland-RL : Multi-Agent Reinforcement Learning on Trains
• Accelerating Training in Pommerman with Imitation and Reinforcement Learning
• Scaling Imitation Learning in Minecraft
• When Is Generalizable Reinforcement Learning Tractable?
• Modularization of End-to-End Learning: Case Study in Arcade Games
• Guaranteeing Reproducibility in Deep Learning Competitions
• Solving Physics Puzzles by Reasoning about Paths
• Winning the ICCV 2019 Learning to Drive Challenge