RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments

Abstract

Exploration in sparse reward environments remains one of the key challenges of model-free reinforcement learning. Instead of solely relying on extrinsic rewards provided by the environment, many state-of-the-art methods use intrinsic rewards to encourage exploration. However, we show that existing methods fall short in procedurally-generated environments where an agent is unlikely to visit a state more than once. We propose a novel type of intrinsic reward which encourages the agent to take actions that lead to significant changes in its learned state representation. We evaluate our method on multiple challenging procedurally-generated tasks in MiniGrid, as well as on tasks with high-dimensional observations used in prior work. Our experiments demonstrate that this approach is more sample efficient than existing exploration methods, particularly for procedurally-generated MiniGrid environments. Furthermore, we analyze the learned behavior as well as the intrinsic reward received by our agent. In contrast to previous approaches, our intrinsic reward does not diminish during the course of training and it rewards the agent substantially more for interacting with objects that it can control.

References in corpus (17)

• Variational Inference with Normalizing Flows
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• State Representation Learning for Control: An Overview
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• A Study on Overfitting in Deep Reinforcement Learning
• Go-Explore: a New Approach for Hard-Exploration Problems
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Assessing Generalization in Deep Reinforcement Learning
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems
• CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant
• ToyBox: Better Atari Environments for Testing Reinforcement Learning Agents

Cited by in corpus (12)

• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• The NetHack Learning Environment
• Focus on Impact: Indoor Exploration with Intrinsic Motivation
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Fast Adaptation via Policy-Dynamics Value Functions
• Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning
• ELLA: Exploration through Learned Language Abstraction
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Adaptive Procedural Task Generation for Hard-Exploration Problems