Composable Planning with Attributes

Abstract

The tasks that an agent will need to solve often are not known during training. However, if the agent knows which properties of the environment are important then, after learning how its actions affect those properties, it may be able to use this knowledge to solve complex tasks without training specifically for them. Towards this end, we consider a setup in which an environment is augmented with a set of user defined attributes that parameterize the features of interest. We propose a method that learns a policy for transitioning between "nearby" sets of attributes, and maintains a graph of possible transitions. Given a task at test time that can be expressed in terms of a target set of attributes, and a current state, our model infers the attributes of the current state and searches over paths through attribute space to get a high level plan, and then uses its low level policy to execute the plan. We show in 3D block stacking, grid-world games, and StarCraft that our model is able to generalize to longer, more complex tasks at test time by composing simpler learned policies.

References in corpus (13)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Gradient Episodic Memory for Continual Learning
• Hindsight Experience Replay
• Modular Multitask Reinforcement Learning with Policy Sketches
• Hybrid Reward Architecture for Reinforcement Learning
• Model-Free Episodic Control
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Neural Episodic Control
• TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Learning to Act by Predicting the Future
• Independently Controllable Factors
• MazeBase: A Sandbox for Learning from Games

Cited by in corpus (17)

• Mapping State Space using Landmarks for Universal Goal Reaching
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Sparse Graphical Memory for Robust Planning
• Modeling Long-horizon Tasks as Sequential Interaction Landscapes
• Learning and Planning with a Semantic Model
• Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
• Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planning
• Learning to Visually Navigate in Photorealistic Environments Without any Supervision
• Bayesian Relational Memory for Semantic Visual Navigation
• Plan2Vec: Unsupervised Representation Learning by Latent Plans
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Evaluating the Apperception Engine
• Visual scoping operations for physical assembly
• Building Intelligent Autonomous Navigation Agents
• Planning with Arithmetic and Geometric Attributes