Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
arXiv:2101.08152
Abstract
Exploration under sparse reward is a long-standing challenge of model-free reinforcement learning. The state-of-the-art methods address this challenge by introducing intrinsic rewards to encourage exploration in novel states or uncertain environment dynamics. Unfortunately, methods based on intrinsic rewards often fall short in procedurally-generated environments, where a different environment is generated in each episode so that the agent is not likely to visit the same state more than once. Motivated by how humans distinguish good exploration behaviors by looking into the entire episode, we introduce RAPID, a simple yet effective episode-level exploration method for procedurally-generated environments. RAPID regards each episode as a whole and gives an episodic exploration score from both per-episode and long-term views. Those highly scored episodes are treated as good exploration behaviors and are stored in a small ranking buffer. The agent then imitates the episodes in the buffer to reproduce the past good exploration behaviors. We demonstrate our method on several procedurally-generated MiniGrid environments, a first-person-view 3D Maze navigation task from MiniWorld, and several sparse MuJoCo tasks. The results show that RAPID significantly outperforms the state-of-the-art intrinsic reward strategies in terms of sample efficiency and final performance. The code is available at https://github.com/daochenzha/rapid
arXiv commentAccepted by ICLR 2021
References in corpus (23)
- Continuous control with deep reinforcement learning
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Deep Reinforcement Learning that Matters
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- A Study on Overfitting in Deep Reinforcement Learning
- A Deeper Look at Experience Replay
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- Model-Free Episodic Control
- Deep Exploration via Randomized Value Functions
- TextWorld: A Learning Environment for Text-based Games
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Revisiting Fundamentals of Experience Replay
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Never Give Up: Learning Directed Exploration Strategies
- The NetHack Learning Environment
- Learning with AMIGo: Adversarially Motivated Intrinsic Goals
- Generative Adversarial Self-Imitation Learning
- RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
- RLCard: A Toolkit for Reinforcement Learning in Card Games
- Learning to Explore with Meta-Policy Gradient
- Training Agents using Upside-Down Reinforcement Learning
- Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Games