Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
arXiv:2010.01062
Abstract
To rapidly learn a new task, it is often essential for agents to explore efficiently -- especially when performance matters from the first timestep. One way to learn such behaviour is via meta-learning. Many existing methods however rely on dense rewards for meta-training, and can fail catastrophically if the rewards are sparse. Without a suitable reward signal, the need for exploration during meta-training is exacerbated. To address this, we propose HyperX, which uses novel reward bonuses for meta-training to explore in approximate hyper-state space (where hyper-states represent the environment state and the agent's task belief). We show empirically that HyperX meta-learns better task-exploration and adapts more successfully to new tasks than existing methods.
arXiv commentPublished at the International Conference on Machine Learning (ICML) 2021
References in corpus (20)
- Auto-Encoding Variational Bayes
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Count-Based Exploration with Neural Density Models
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- Deep Exploration via Randomized Value Functions
- Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
- Some Considerations on Learning to Explore via Meta-Reinforcement Learning
- VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
- Meta reinforcement learning as task inference
- Uncertainty in Neural Networks: Approximately Bayesian Ensembling
- Meta-learning of Sequential Strategies
- Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization
- Offline Meta Learning of Exploration
- MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
- Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
- MAME : Model-Agnostic Meta-Exploration