A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning

Abstract

Across machine learning, the use of curricula has shown strong empirical potential to improve learning from data by avoiding local optima of training objectives. For reinforcement learning (RL), curricula are especially interesting, as the underlying optimization has a strong tendency to get stuck in local optima due to the exploration-exploitation trade-off. Recently, a number of approaches for an automatic generation of curricula for RL have been shown to increase performance while requiring less expert knowledge compared to manually designed curricula. However, these approaches are seldomly investigated from a theoretical perspective, preventing a deeper understanding of their mechanics. In this paper, we present an approach for automated curriculum generation in RL with a clear theoretical underpinning. More precisely, we formalize the well-known self-paced learning paradigm as inducing a distribution over training tasks, which trades off between task complexity and the objective to match a desired task distribution. Experiments show that training on this induced distribution helps to avoid poor local optima across RL algorithms in different tasks with uninformative rewards and challenging exploration requirements.

References in corpus (18)

• Nearly unbiased variable selection under minimax concave penalty
• Trust Region Policy Optimization
• Simulated Tempering: A New Monte Carlo Scheme
• VIME: Variational Information Maximizing Exploration
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Hindsight Experience Replay
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Safe Controller Optimization for Quadrotors with Gaussian Processes
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Unifying Count-Based Exploration and Intrinsic Motivation
• Maximum a Posteriori Policy Optimisation
• Reverse Curriculum Generation for Reinforcement Learning
• Understanding and Accelerating Particle-Based Variational Inference
• VIREL: A Variational Inference Framework for Reinforcement Learning
• Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem
• Variational Tempering
• High Acceleration Reinforcement Learning for Real-World Juggling with Binary Rewards