State Entropy Maximization with Random Encoders for Efficient Exploration

Abstract

Recent exploration methods have proven to be a recipe for improving sample-efficiency in deep reinforcement learning (RL). However, efficient exploration in high-dimensional observation spaces still remains a challenge. This paper presents Random Encoders for Efficient Exploration (RE3), an exploration method that utilizes state entropy as an intrinsic reward. In order to estimate state entropy in environments with high-dimensional observations, we utilize a k-nearest neighbor entropy estimator in the low-dimensional representation space of a convolutional encoder. In particular, we find that the state entropy can be estimated in a stable and compute-efficient manner by utilizing a randomly initialized encoder, which is fixed throughout training. Our experiments show that RE3 significantly improves the sample-efficiency of both model-free and model-based RL methods on locomotion and navigation tasks from DeepMind Control Suite and MiniGrid benchmarks. We also show that RE3 allows learning diverse behaviors without extrinsic rewards, effectively improving sample-efficiency in downstream tasks. Source code and videos are available at https://sites.google.com/view/re3-rl.

arXiv commentICML 2021. First two authors contributed equally. Website: https://sites.google.com/view/re3-rl Code: https://github.com/younggyoseo/RE3

References in corpus (17)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Noisy Networks for Exploration
• VIME: Variational Information Maximizing Exploration
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Visual Reinforcement Learning with Imagined Goals
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• dm_control: Software and Tasks for Continuous Control
• Provably Efficient Maximum Entropy Exploration
• Efficient Exploration via State Marginal Matching
• Never Give Up: Learning Directed Exploration Strategies
• No Training Required: Exploring Random Encoders for Sentence Classification
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• Behavior From the Void: Unsupervised Active Pre-Training
• Training CNNs with Selective Allocation of Channels

Cited by in corpus (10)

• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• Maximum diffusion reinforcement learning
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning
• Reinforcement Learning by Guided Safe Exploration
• Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning
• The Information Geometry of Unsupervised Reinforcement Learning
• Making Curiosity Explicit in Vision-based RL
• Learning Altruistic Behaviours in Reinforcement Learning without External Rewards