BeBold: Exploration Beyond the Boundary of Explored Regions

Abstract

Efficient exploration under sparse rewards remains a key challenge in deep reinforcement learning. To guide exploration, previous work makes extensive use of intrinsic reward (IR). There are many heuristics for IR, including visitation counts, curiosity, and state-difference. In this paper, we analyze the pros and cons of each method and propose the regulated difference of inverse visitation counts as a simple but effective criterion for IR. The criterion helps the agent explore Beyond the Boundary of explored regions and mitigates common issues in count-based methods, such as short-sightedness and detachment. The resulting method, BeBold, solves the 12 most challenging procedurally-generated tasks in MiniGrid with just 120M environment steps, without any curriculum learning. In comparison, the previous SoTA only solves 50% of the tasks. BeBold also achieves SoTA on multiple tasks in NetHack, a popular rogue-like game that contains more challenging procedurally-generated environments.

References in corpus (27)

• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• Noisy Networks for Exploration
• Parameter Space Noise for Exploration
• Large-Scale Study of Curiosity-Driven Learning
• Count-Based Exploration with Neural Density Models
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Exploration by Random Network Distillation
• Agent57: Outperforming the Atari Human Benchmark
• Go-Explore: a New Approach for Hard-Exploration Problems
• Variational Intrinsic Control
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Deep Exploration via Randomized Value Functions
• Reverse Curriculum Generation for Reinforcement Learning
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• Efficient Exploration via State Marginal Matching
• Model-Based Active Exploration
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• The NetHack Learning Environment
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Unsupervised Curricula for Visual Meta-Reinforcement Learning
• Frontier Based Exploration for Autonomous Robot
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• Self-Supervised Exploration via Disagreement
• Directed Exploration for Reinforcement Learning

Cited by in corpus (9)

• MADE: Exploration via Maximizing Deviation from Explored Regions
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
• BIMRL: Brain Inspired Meta Reinforcement Learning
• Prioritized Level Replay
• C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
• Co-Imitation Learning without Expert Demonstration
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• Generalization in Text-based Games via Hierarchical Reinforcement Learning