Watch, Try, Learn: Meta-Learning from Demonstrations and Reward

Abstract

Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging experience from learning similar tasks. In the presence of task ambiguity or unobserved dynamics, demonstrations alone may not provide enough information; an agent must also try the task to successfully infer a policy. In this work, we propose a method that can learn to learn from both demonstrations and trial-and-error experience with sparse reward feedback. In comparison to meta-imitation, this approach enables the agent to effectively and efficiently improve itself autonomously beyond the demonstration data. In comparison to meta-reinforcement learning, we can scale to substantially broader distributions of tasks, as the demonstration reduces the burden of exploration. Our experiments show that our method significantly outperforms prior approaches on a set of challenging, vision-based control tasks.

References in corpus (17)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Theoretical Models of Learning to Learn
• A Simple Neural Attentive Meta-Learner
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Continuous Deep Q-Learning with Model-based Acceleration
• Deep Q-learning from Demonstrations
• One-Shot Visual Imitation Learning via Meta-Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Evolved Policy Gradients
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Hierarchical Imitation and Reinforcement Learning
• One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL

Cited by in corpus (9)

• BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Transformers for One-Shot Visual Imitation
• A Mobile Manipulation System for One-Shot Teaching of Complex Tasks in Homes
• Guided Exploration with Proximal Policy Optimization using a Single Demonstration
• Visual Goal-Directed Meta-Learning with Contextual Planning Networks
• Curiosity-driven Intuitive Physics Learning