TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
arXiv:2103.09815
Abstract
Training autonomous agents able to generalize to multiple tasks is a key target of Deep Reinforcement Learning (DRL) research. In parallel to improving DRL algorithms themselves, Automatic Curriculum Learning (ACL) study how teacher algorithms can train DRL agents more efficiently by adapting task selection to their evolving abilities. While multiple standard benchmarks exist to compare DRL agents, there is currently no such thing for ACL algorithms. Thus, comparing existing approaches is difficult, as too many experimental parameters differ from paper to paper. In this work, we identify several key challenges faced by ACL algorithms. Based on these, we present TeachMyAgent (TA), a benchmark of current ACL algorithms leveraging procedural task generation. It includes 1) challenge-specific unit-tests using variants of a procedural Box2D bipedal walker environment, and 2) a new procedural Parkour environment combining most ACL challenges, making it ideal for global performance assessment. We then use TeachMyAgent to conduct a comparative study of representative existing approaches, showcasing the competitiveness of some ACL algorithms that do not use expert knowledge. We also show that the Parkour environment remains an open problem. We open-source our environments, all studied ACL algorithms (collected from open-source code or re-implemented), and DRL students in a Python package available at https://github.com/flowersteam/TeachMyAgent.
References in corpus (19)
- Continuous control with deep reinforcement learning
- Overcoming catastrophic forgetting in neural networks
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Meta-Learning: A Survey
- Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
- Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- TextWorld: A Learning Environment for Text-based Games
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Learning Montezuma's Revenge from a Single Demonstration
- BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
- How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
- Learning with AMIGo: Adversarially Motivated Intrinsic Goals
- Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
- Automatic Curriculum Learning through Value Disagreement
- Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challenge
- Automated curricula through setter-solver interactions
- Meta Automatic Curriculum Learning
- Prioritized Level Replay