A Brief Look at Generalization in Visual Meta-Reinforcement Learning
arXiv:2006.07262
Abstract
Due to the realization that deep reinforcement learning algorithms trained on high-dimensional tasks can strongly overfit to their training environments, there have been several studies that investigated the generalization performance of these algorithms. However, there has been no similar study that evaluated the generalization performance of algorithms that were specifically designed for generalization, i.e. meta-reinforcement learning algorithms. In this paper, we assess the generalization performance of these algorithms by leveraging high-dimensional, procedurally generated environments. We find that these algorithms can display strong overfitting when they are evaluated on challenging tasks. We also observe that scalability to high-dimensional tasks with sparse rewards remains a significant problem among many of the current meta-reinforcement learning algorithms. With these results, we highlight the need for developing meta-reinforcement learning algorithms that can both generalize and scale.
Accepted to the 4th Lifelong Learning Workshop at ICML 2020
References in corpus (15)
- StarCraft II: A New Challenge for Reinforcement Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Learning to reinforcement learn
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- A Study on Overfitting in Deep Reinforcement Learning
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- Quantifying Generalization in Reinforcement Learning
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- Generalization and Regularization in DQN
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Some Considerations on Learning to Explore via Meta-Reinforcement Learning
- Improving Generalization in Meta Reinforcement Learning using Learned Objectives