Self-Paced Deep Reinforcement Learning
arXiv:2004.11812
Abstract
Curriculum reinforcement learning (CRL) improves the learning speed and stability of an agent by exposing it to a tailored series of tasks throughout learning. Despite empirical successes, an open question in CRL is how to automatically generate a curriculum for a given reinforcement learning (RL) agent, avoiding manual design. In this paper, we propose an answer by interpreting the curriculum generation as an inference problem, where distributions over tasks are progressively learned to approach the target task. This approach leads to an automatic curriculum generation, whose pace is controlled by the agent, with solid theoretical motivation and easily integrated with deep RL algorithms. In the conducted experiments, the curricula generated with the proposed algorithm significantly improve learning performance across several environments and deep RL algorithms, matching or outperforming state-of-the-art existing CRL algorithms.
References in corpus (14)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- VIME: Variational Information Maximizing Exploration
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Hindsight Experience Replay
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Learning by Playing - Solving Sparse Reward Tasks from Scratch
- Unifying Count-Based Exploration and Intrinsic Motivation
- Maximum a Posteriori Policy Optimisation
- Understanding and Accelerating Particle-Based Variational Inference
- VIREL: A Variational Inference Framework for Reinforcement Learning
- Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem
- Variational Tempering
- Self-Paced Contextual Reinforcement Learning