C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks

Abstract

Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range of domains, including navigation and manipulation, but learning to reach distant goals remains a central challenge to the field. Learning to reach such goals is particularly hard without any offline data, expert demonstrations, and reward shaping. In this paper, we propose an algorithm to solve the distant goal-reaching task by using search at training time to automatically generate a curriculum of intermediate states. Our algorithm, Classifier-Planning (C-Planning), frames the learning of the goal-conditioned policies as expectation maximization: the E-step corresponds to planning an optimal sequence of waypoints using graph search, while the M-step aims to learn a goal-conditioned policy to reach those waypoints. Unlike prior methods that combine goal-conditioned RL with graph search, ours performs search only during training and not testing, significantly decreasing the compute costs of deploying the learned policy. Empirically, we demonstrate that our method is more sample efficient than prior methods. Moreover, it is able to solve very long horizons manipulation and navigation tasks, tasks that prior goal-conditioned methods and methods based on graph search fail to solve.

References in corpus (9)

• Hyperbolic Discounting and Learning over Multiple Horizons
• Planning with Goal-Conditioned Policies
• Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Goal-Conditioned Reinforcement Learning with Imagined Subgoals
• BeBold: Exploration Beyond the Boundary of Explored Regions
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Reinforcement Learning without Ground-Truth State
• Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning