MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
arXiv:2006.08170
Abstract
Meta reinforcement learning (meta-RL) extracts knowledge from previous tasks and achieves fast adaptation to new tasks. Despite recent progress, efficient exploration in meta-RL remains a key challenge in sparse-reward tasks, as it requires quickly finding informative task-relevant experiences in both meta-training and adaptation. To address this challenge, we explicitly model an exploration policy learning problem for meta-RL, which is separated from exploitation policy learning, and introduce a novel empowerment-driven exploration objective, which aims to maximize information gain for task identification. We derive a corresponding intrinsic reward and develop a new off-policy meta-RL framework, which efficiently learns separate context-aware exploration and exploitation policies by sharing the knowledge of task inference. Experimental evaluation shows that our meta-RL method significantly outperforms state-of-the-art baselines on various sparse-reward MuJoCo locomotion tasks and more complex sparse-reward Meta-World tasks.
References in corpus (21)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Theoretical Models of Learning to Learn
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- A Simple Neural Attentive Meta-Learner
- A Recurrent Latent Variable Model for Sequential Data
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- VIME: Variational Information Maximizing Exploration
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Deep Variational Information Bottleneck
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- (More) Efficient Reinforcement Learning via Posterior Sampling
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
- Meta reinforcement learning as task inference
- Unsupervised Meta-Learning for Reinforcement Learning
- Environment Probing Interaction Policies
- EMI: Exploration with Mutual Information
- Thompson Sampling is Asymptotically Optimal in General Environments