Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
arXiv:1910.04142
Abstract
Humans are masters at quickly learning many complex tasks, relying on an approximate understanding of the dynamics of their environments. In much the same way, we would like our learning agents to quickly adapt to new tasks. In this paper, we explore how model-based Reinforcement Learning (RL) can facilitate transfer to new tasks. We develop an algorithm that learns an action-conditional, predictive model of expected future observations, rewards and values from which a policy can be derived by following the gradient of the estimated value along imagined trajectories. We show how robust policy optimization can be achieved in robot manipulation tasks even with approximate models that are learned directly from vision and proprioception. We evaluate the efficacy of our approach in a transfer learning scenario, re-using previously learned models on tasks with different reward structures and visual distractors, and show a significant improvement in learning speed compared to strong off-policy baselines. Videos with results can be found at https://sites.google.com/view/ivg-corl19
To appear at the 3rd annual Conference on Robot Learning, Osaka, Japan (CoRL 2019). 24 pages including appendix (main paper - 8 pages)
References in corpus (26)
- Auto-Encoding Variational Bayes
- Continuous control with deep reinforcement learning
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Soft Actor-Critic Algorithms and Applications
- Wide Residual Networks
- Emergence of Locomotion Behaviours in Rich Environments
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Model-Based Reinforcement Learning for Atari
- Learning Latent Dynamics for Planning from Pixels
- Continuous Deep Q-Learning with Model-based Acceleration
- Learning Dexterous In-Hand Manipulation
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- Maximum a Posteriori Policy Optimisation
- Imagination-Augmented Agents for Deep Reinforcement Learning
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
- Unsupervised Predictive Memory in a Goal-Directed Agent
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
- Relative Entropy Regularized Policy Iteration
- Information asymmetry in KL-regularized RL
- Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
- The Devil is in the Decoder: Classification, Regression and GANs