Automatic Curriculum Learning through Value Disagreement

Abstract

Continually solving new, unsolved tasks is the key to learning diverse behaviors. Through reinforcement learning (RL), we have made massive strides towards solving tasks that have a single goal. However, in the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. When biological agents learn, there is often an organized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve. Our key insight is that if we can sample goals at the frontier of the set of goals that an agent is able to reach, it will provide a significantly stronger learning signal compared to randomly sampled goals. To operationalize this idea, we introduce a goal proposal module that prioritizes goals that maximize the epistemic uncertainty of the Q-function of the policy. This simple technique samples goals that are neither too hard nor too easy for the agent to solve, hence enabling continual improvement. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.

https://sites.google.com/berkeley.edu/vds/

References in corpus (15)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Dota 2 with Large Scale Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Robust Adversarial Reinforcement Learning
• Emergent Tool Use From Multi-Agent Autocurricula
• Emergent Complexity via Multi-Agent Competition
• Automatic Goal Generation for Reinforcement Learning Agents
• Reverse Curriculum Generation for Reinforcement Learning
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Deep Drone Racing: Learning Agile Flight in Dynamic Environments
• Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
• Self-Supervised Exploration via Disagreement

Cited by in corpus (9)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Dealing with Sparse Rewards Using Graph Neural Networks
• Adversarial Intrinsic Motivation for Reinforcement Learning
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learning
• Automatic Goal Generation using Dynamical Distance Learning