Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning

Abstract

In hierarchical reinforcement learning a major challenge is determining appropriate low-level policies. We propose an unsupervised learning scheme, based on asymmetric self-play from Sukhbaatar et al. (2018), that automatically learns a good representation of sub-goals in the environment and a low-level policy that can execute them. A high-level policy can then direct the lower one by generating a sequence of continuous sub-goal vectors. We evaluate our model using Mazebase and Mujoco environments, including the challenging AntGather task. Visualizations of the sub-goal embeddings reveal a logical decomposition of tasks within the environment. Quantitatively, our approach obtains compelling performance gains over non-hierarchical approaches.

References in corpus (6)

• Benchmarking Deep Reinforcement Learning for Continuous Control
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Variational Intrinsic Control
• Reverse Curriculum Generation for Reinforcement Learning
• Latent Space Policies for Hierarchical Reinforcement Learning
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

Cited by in corpus (10)

• A survey on intrinsic motivation in reinforcement learning
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
• Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft
• Learning to Prove Theorems by Learning to Generate Theorems
• Augmenting GAIL with BC for sample efficient imitation learning
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms