Universal Planning Networks

Abstract

A key challenge in complex visuomotor control is learning abstract representations that are effective for specifying goals, planning, and generalization. To this end, we introduce universal planning networks (UPN). UPNs embed differentiable planning within a goal-directed policy. This planning computation unrolls a forward model in a latent space and infers an optimal action plan through gradient descent trajectory optimization. The plan-by-gradient-descent process and its underlying representations are learned end-to-end to directly optimize a supervised imitation learning objective. We find that the representations learned are not only effective for goal-directed visual imitation via gradient-based trajectory optimization, but can also provide a metric for specifying goals using images. The learned representations can be leveraged to specify distance-based rewards to reach new target states for model-free reinforcement learning, resulting in substantially more effective learning when solving new tasks described via image-based goals. We were able to achieve successful transfer of visuomotor planning strategies across robots with significantly different morphologies and actuation capabilities.

arXiv commentVideos available at https://sites.google.com/view/upn-public/home

References in corpus (25)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• End-to-End Training of Deep Visuomotor Policies
• Searching for Activation Functions
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Generative Adversarial Imitation Learning
• Hindsight Experience Replay
• One-Shot Visual Imitation Learning via Meta-Learning
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• The Predictron: End-To-End Learning and Planning
• Emergent Complexity via Multi-Agent Competition
• OptNet: Differentiable Optimization as a Layer in Neural Networks
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• PixelSNAIL: An Improved Autoregressive Generative Model
• Time-Contrastive Networks: Self-Supervised Learning from Video
• Value Prediction Network
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
• MPC-Inspired Neural Network Policies for Sequential Decision Making
• TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning
• Incorporating Human Domain Knowledge into Large Scale Cost Function Learning

Cited by in corpus (43)

• Visual Reinforcement Learning with Imagined Goals
• Differentiable MPC for End-to-end Planning and Control
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
• Dream to Control: Learning Behaviors by Latent Imagination
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Unsupervised Meta-Learning for Reinforcement Learning
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Model-Augmented Actor-Critic: Backpropagating through Paths
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• SPNets: Differentiable Fluid Dynamics for Deep Neural Networks
• Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight
• Risks from Learned Optimization in Advanced Machine Learning Systems
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
• Bayesian policy selection using active inference
• Mastering Atari with Discrete World Models
• World Model as a Graph: Learning Latent Landmarks for Planning
• Objective Mismatch in Model-based Reinforcement Learning
• Combining Q-Learning and Search with Amortized Value Estimates
• On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference
• Propagation Networks for Model-Based Control Under Partial Observation
• Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
• Procedure Planning in Instructional Videos
• Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
• To Follow or not to Follow: Selective Imitation Learning from Observations
• Deep Residual Reinforcement Learning
• The Differentiable Cross-Entropy Method
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Robot Motion Planning in Learned Latent Spaces
• Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation
• NOVAS: Non-convex Optimization via Adaptive Stochastic Search for End-to-End Learning and Control
• Probably Approximately Correct Vision-Based Planning using Motion Primitives
• PathBench: A Benchmarking Platform for Classical and Learned Path Planning Algorithms
• Hierarchies of Planning and Reinforcement Learning for Robot Navigation
• LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
• Robotic self-representation improves manipulation skills and transfer learning
• Reinforcement Learning as Iterative and Amortised Inference
• Learning to Predict Ego-Vehicle Poses for Sampling-Based Nonholonomic Motion Planning
• Visual Goal-Directed Meta-Learning with Contextual Planning Networks
• Waypoint Planning Networks
• Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning