MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration

Abstract

Meta reinforcement learning (meta-RL) extracts knowledge from previous tasks and achieves fast adaptation to new tasks. Despite recent progress, efficient exploration in meta-RL remains a key challenge in sparse-reward tasks, as it requires quickly finding informative task-relevant experiences in both meta-training and adaptation. To address this challenge, we explicitly model an exploration policy learning problem for meta-RL, which is separated from exploitation policy learning, and introduce a novel empowerment-driven exploration objective, which aims to maximize information gain for task identification. We derive a corresponding intrinsic reward and develop a new off-policy meta-RL framework, which efficiently learns separate context-aware exploration and exploitation policies by sharing the knowledge of task inference. Experimental evaluation shows that our meta-RL method significantly outperforms state-of-the-art baselines on various sparse-reward MuJoCo locomotion tasks and more complex sparse-reward Meta-World tasks.

References in corpus (7)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Theoretical Models of Learning to Learn
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

Cited by in corpus (4)

• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• Explore and Control with Adversarial Surprise
• HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem