HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem

Abstract

In spite of the success of existing meta reinforcement learning methods, they still have difficulty in learning a meta policy effectively for RL problems with sparse reward. In this respect, we develop a novel meta reinforcement learning framework called Hyper-Meta RL(HMRL), for sparse reward RL problems. It is consisted with three modules including the cross-environment meta state embedding module which constructs a common meta state space to adapt to different environments; the meta state based environment-specific meta reward shaping which effectively extends the original sparse reward trajectory by cross-environmental knowledge complementarity and as a consequence the meta policy achieves better generalization and efficiency with the shaped meta reward. Experiments with sparse-reward environments show the superiority of HMRL on both transferability and policy learning efficiency.

13 pages

References in corpus (12)

• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Dealing with Sparse Rewards in Reinforcement Learning
• Reward Shaping via Meta-Learning
• Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Meta Learning MPC using Finite-Dimensional Gaussian Process Approximations
• MAME : Model-Agnostic Meta-Exploration