Adaptive Procedural Task Generation for Hard-Exploration Problems
arXiv:2007.00350
Abstract
We introduce Adaptive Procedural Task Generation (APT-Gen), an approach to progressively generate a sequence of tasks as curricula to facilitate reinforcement learning in hard-exploration problems. At the heart of our approach, a task generator learns to create tasks from a parameterized task space via a black-box procedural generation module. To enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balancing the agent's performance in the generated tasks and the similarity to the target tasks. Through adversarial training, the task similarity is adaptively estimated by a task discriminator defined on the agent's experiences, allowing the generated tasks to approximate target tasks of unknown parameterization or outside of the predefined task space. Our experiments on the grid world and robotic manipulation task domains show that APT-Gen achieves substantially better performance than various existing baselines by generating suitable tasks of rich variations.
arXiv commentICLR 2021
References in corpus (31)
- Adam: A Method for Stochastic Optimization
- Overcoming catastrophic forgetting in neural networks
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Playing Atari with Deep Reinforcement Learning
- Solving Rubik's Cube with a Robot Hand
- VIME: Variational Information Maximizing Exploration
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Emergent Tool Use From Multi-Agent Autocurricula
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- AI2-THOR: An Interactive 3D Environment for Visual AI
- Reinforcement Learning with Unsupervised Auxiliary Tasks
- Learning by Playing - Solving Sparse Reward Tasks from Scratch
- Agent57: Outperforming the Atari Human Benchmark
- Go-Explore: a New Approach for Hard-Exploration Problems
- Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
- Procedural Content Generation through Quality Diversity
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- Learning Montezuma's Revenge from a Single Demonstration
- Learning to Infer and Execute 3D Shape Programs
- Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
- Unsupervised Meta-Learning for Reinforcement Learning
- Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
- RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
- Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
- Mix&Match - Agent Curricula for Reinforcement Learning
- One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
- Unsupervised Curricula for Visual Meta-Reinforcement Learning
- Situational Fusion of Visual Representation for Visual Navigation
- Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
- PDDLGym: Gym Environments from PDDL Problems