Meta Automatic Curriculum Learning
arXiv:2011.08463
Abstract
A major challenge in the Deep RL (DRL) community is to train agents able to generalize their control policy over situations never seen in training. Training on diverse tasks has been identified as a key ingredient for good generalization, which pushed researchers towards using rich procedural task generation systems controlled through complex continuous parameter spaces. In such complex task spaces, it is essential to rely on some form of Automatic Curriculum Learning (ACL) to adapt the task sampling distribution to a given learning agent, instead of randomly sampling tasks, as many could end up being either trivial or unfeasible. Since it is hard to get prior knowledge on such task spaces, many ACL algorithms explore the task space to detect progress niches over time, a costly tabula-rasa process that needs to be performed for each new learning agents, although they might have similarities in their capabilities profiles. To address this limitation, we introduce the concept of Meta-ACL, and formalize it in the context of black-box RL learners, i.e. algorithms seeking to generalize curriculum generation to an (unknown) distribution of learners. In this work, we present AGAIN, a first instantiation of Meta-ACL, and showcase its benefits for curriculum generation over classical ACL in multiple simulated environments including procedurally generated parkour environments with learners of varying morphologies. Videos and code are available at https://sites.google.com/view/meta-acl .
arXiv commentThis paper extends and generalizes work in arXiv:2004.03168
References in corpus (17)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Solving Rubik's Cube with a Robot Hand
- Learning to reinforcement learn
- Hindsight Experience Replay
- Meta-Learning: A Survey
- Distral: Robust Multitask Reinforcement Learning
- Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
- Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- Multi-Armed Bandits for Intelligent Tutoring Systems
- Learning Montezuma's Revenge from a Single Demonstration
- Safe Reinforcement Learning via Curriculum Induction
- Automated curricula through setter-solver interactions
- Unsupervised Curricula for Visual Meta-Reinforcement Learning
- Accuracy-based Curriculum Learning in Deep Reinforcement Learning