Planning with Goal-Conditioned Policies

Abstract

Planning methods can solve temporally extended sequential decision making problems by composing simple behaviors. However, planning requires suitable abstractions for the states and transitions, which typically need to be designed by hand. In contrast, model-free reinforcement learning (RL) can acquire behaviors from low-level inputs directly, but often struggles with temporally extended tasks. Can we utilize reinforcement learning to automatically form the abstractions needed for planning, thus obtaining the best of both approaches? We show that goal-conditioned policies learned with RL can be incorporated into planning, so that a planner can focus on which states to reach, rather than how those states are reached. However, with complex state observations such as images, not all inputs represent valid states. We therefore also propose using a latent variable model to compactly represent the set of valid states for the planner, so that the policies provide an abstraction of actions, and the latent variable model provides an abstraction of states. We compare our method with planning-based and model-free methods and find that our method significantly outperforms prior work when evaluated on image-based robot navigation and manipulation tasks that require non-greedy, multi-staged behavior.

arXiv commentIn Advances in Neural Information Processing Systems, 2019

Cited by in corpus (26)

• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
• World Model as a Graph: Learning Latent Landmarks for Planning
• PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
• Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planning
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• TempoRL: Learning When to Act
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Deep Reactive Planning in Dynamic Environments
• Implicit Finite-Horizon Approximation and Efficient Optimal Algorithms for Stochastic Shortest Path
• C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
• Hierarchical Robot Navigation in Novel Environments using Rough 2-D Maps
• Adaptable Agent Populations via a Generative Model of Policies
• Goal Kernel Planning: Linearly-Solvable Non-Markovian Policies for Logical Tasks with Goal-Conditioned Options
• Self-supervised Visual Reinforcement Learning with Object-centric Representations
• Goal-conditioned Batch Reinforcement Learning for Rotation Invariant Locomotion
• Learning Stochastic Shortest Path with Linear Function Approximation
• Multi-task Reinforcement Learning with a Planning Quasi-Metric
• Manipulator-Independent Representations for Visual Imitation
• Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph
• Learning Synthetic to Real Transfer for Localization and Navigational Tasks