MELD: Meta-Reinforcement Learning from Images via Latent State Models
arXiv:2010.13957
Abstract
Meta-reinforcement learning algorithms can enable autonomous agents, such as robots, to quickly acquire new behaviors by leveraging prior experience in a set of related training tasks. However, the onerous data requirements of meta-training compounded with the challenge of learning from sensory inputs such as images have made meta-RL challenging to apply to real robotic systems. Latent state models, which learn compact state representations from a sequence of observations, can accelerate representation learning from visual inputs. In this paper, we leverage the perspective of meta-learning as task inference to show that latent state models can \emph{also} perform meta-learning given an appropriately defined observation space. Building on this insight, we develop meta-RL with latent dynamics (MELD), an algorithm for meta-RL from images that performs inference in a latent state model to quickly acquire new skills given observations and rewards. MELD outperforms prior meta-RL methods on several simulated image-based robotic control problems, and enables a real WidowX robotic arm to insert an Ethernet cable into new locations given a sparse task completion signal after only hours of real world meta-training. To our knowledge, MELD is the first meta-RL algorithm trained in a real-world robotic control setting from images.
arXiv commentAccepted to CoRL 2020. Supplementary material at https://sites.google.com/view/meld-lsm/home . 16 pages, 19 figures. V2: add funding acknowledgements, reduce file size
References in corpus (23)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- A Simple Neural Attentive Meta-Learner
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Learning to reinforcement learn
- Learning Latent Dynamics for Planning from Pixels
- One-Shot Visual Imitation Learning via Meta-Learning
- Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- Memory-based control with recurrent neural networks
- Evolved Policy Gradients
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
- VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
- Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
- Meta reinforcement learning as task inference
- Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
- Few-Shot Goal Inference for Visuomotor Learning and Planning
- Task-Embedded Control Networks for Few-Shot Imitation Learning
- Temporal Difference Variational Auto-Encoder
- Contextual Reinforcement Learning of Visuo-tactile Multi-fingered Grasping Policies
- Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials