#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

Abstract

Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs). It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once. Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation. In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various high-dimensional and/or continuous deep RL benchmarks. States are mapped to hash codes, which allows to count their occurrences with a hash table. These counts are then used to compute a reward bonus according to the classic count-based exploration theory. We find that simple hash functions can achieve surprisingly good results on many challenging tasks. Furthermore, we show that a domain-dependent learned hash code may further improve these results. Detailed analysis reveals important aspects of a good hash function: 1) having appropriate granularity and 2) encoding information relevant to solving the MDP. This exploration strategy achieves near state-of-the-art performance on both continuous control tasks and Atari 2600 games, hence providing a simple yet powerful baseline for solving MDPs that require considerable exploration.

arXiv comment10 pages main text + 10 pages supplementary. Published at NIPS 2017

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Continuous control with deep reinforcement learning
• Deep Residual Learning for Image Recognition
• Trust Region Policy Optimization
• Dueling Network Architectures for Deep Reinforcement Learning
• Pixel Recurrent Neural Networks
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Deep Exploration via Bootstrapped DQN
• Massively Parallel Methods for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Bayesian Reinforcement Learning: A Survey

Cited by in corpus (151)

• Deep Reinforcement Learning: An Overview
• Emergent Tool Use From Multi-Agent Autocurricula
• Large-Scale Study of Curiosity-Driven Learning
• Exploration by Random Network Distillation
• Agent57: Outperforming the Atari Human Benchmark
• First return, then explore
• Go-Explore: a New Approach for Hard-Exploration Problems
• Episodic Curiosity through Reachability
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• A Survey of Deep Reinforcement Learning in Video Games
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Reinforcement Learning in Healthcare: A Survey
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• A survey on intrinsic motivation in reinforcement learning
• Efficient Exploration via State Marginal Matching
• Learning Montezuma's Revenge from a Single Demonstration
• UCB Exploration via Q-Ensembles
• Dynamics-Aware Unsupervised Discovery of Skills
• Contingency-Aware Exploration in Reinforcement Learning
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• Optimism in Reinforcement Learning with Generalized Linear Function Approximation
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• The NetHack Learning Environment
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• On Reward-Free Reinforcement Learning with Linear Function Approximation
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• On Learning Intrinsic Rewards for Policy Gradient Methods
• An Adaptive Clipping Approach for Proximal Policy Optimization
• Learning to Explore with Meta-Policy Gradient
• Learning Self-Imitating Diverse Policies
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• Reinforcement Learning under Threats
• On Bonus-Based Exploration Methods in the Arcade Learning Environment
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search
• Learning latent state representation for speeding up exploration
• Efficient Model-free Reinforcement Learning in Metric Spaces
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• See, Hear, Explore: Curiosity via Audio-Visual Association
• Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
• Meta-learning curiosity algorithms
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
• Parallel Exploration via Negatively Correlated Search
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Optimistic Exploration even with a Pessimistic Initialisation
• Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization
• Revisiting Design Choices in Proximal Policy Optimization
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Adaptive Reward-Free Exploration
• Directed Exploration for Reinforcement Learning
• Never Forget: Balancing Exploration and Exploitation via Learning Optical Flow
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning
• Self-Imitation Advantage Learning
• Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks
• MULEX: Disentangling Exploitation from Exploration in Deep RL
• Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning
• Online Learning in Contextual Bandits using Gated Linear Networks
• Bayesian Curiosity for Efficient Exploration in Reinforcement Learning
• Principled Exploration via Optimistic Bootstrapping and Backward Induction
• Marginalized State Distribution Entropy Regularization in Policy Optimization
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Exploring Exploration: Comparing Children with RL Agents in Unified Environments
• Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation
• An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environments
• The Laplacian in RL: Learning Representations with Efficient Approximations
• Reinforcement Learning, Bit by Bit
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Influence-Based Multi-Agent Exploration
• Model Based Planning with Energy Based Models
• Imitation Learning from Pixel-Level Demonstrations by HashReward
• Exploration Bonus for Regret Minimization in Undiscounted Discrete and Continuous Markov Decision Processes
• Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning
• Zero-Shot Learning of Text Adventure Games with Sentence-Level Semantics
• Competitive Experience Replay
• General Policies, Serializations, and Planning Width
• Multi-Path Policy Optimization
• Fast Exploration with Simplified Models and Approximately Optimistic Planning in Model Based Reinforcement Learning
• Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• Learning Abstract Models for Strategic Exploration and Fast Reward Transfer
• REMAX: Relational Representation for Multi-Agent Exploration
• Beyond Exponentially Discounted Sum: Automatic Learning of Return Function
• Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards
• Continuous Control with Action Quantization from Demonstrations
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• Directed Exploration in PAC Model-Free Reinforcement Learning
• An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards
• Provably Efficient Exploration for Reinforcement Learning Using Unsupervised Learning
• Goal-oriented Trajectories for Efficient Exploration
• Guided Exploration with Proximal Policy Optimization using a Single Demonstration
• Improved Exploring Starts by Kernel Density Estimation-Based State-Space Coverage Acceleration in Reinforcement Learning
• Policy Optimization with Model-based Explorations
• TempLe: Learning Template of Transitions for Sample Efficient Multi-task RL
• Novel Policy Seeking with Constrained Optimization
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• TTR-Based Reward for Reinforcement Learning with Implicit Model Priors
• Hierarchical Skills for Efficient Exploration
• A Short Survey on Probabilistic Reinforcement Learning
• Regularly Updated Deterministic Policy Gradient Algorithm
• Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks
• Potential-Based Advice for Stochastic Policy Learning
• Towards Combining On-Off-Policy Methods for Real-World Applications
• Exploring Unknown States with Action Balance
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Clustered Reinforcement Learning
• TAMPC: A Controller for Escaping Traps in Novel Environments
• Generative Adversarial Exploration for Reinforcement Learning
• Towards Tractable Optimism in Model-Based Reinforcement Learning
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• Importance Sampling based Exploration in Q Learning
• SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Reannealing of Decaying Exploration Based On Heuristic Measure in Deep Q-Network
• ADER:Adapting between Exploration and Robustness for Actor-Critic Methods
• Provable RL with Exogenous Distractors via Multistep Inverse Dynamics
• Self-Paced Deep Reinforcement Learning
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Optimising Stochastic Routing for Taxi Fleets with Model Enhanced Reinforcement Learning
• Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning
• A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning
• Hyperspace Neighbor Penetration Approach to Dynamic Programming for Model-Based Reinforcement Learning Problems with Slowly Changing Variables in A Continuous State Space
• Option Discovery in the Absence of Rewards with Manifold Analysis
• Uncertainty-aware Low-Rank Q-Matrix Estimation for Deep Reinforcement Learning
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
• Bayesian Distributional Policy Gradients
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Knowledge is reward: Learning optimal exploration by predictive reward cashing
• Evaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
• Generative Exploration and Exploitation
• Exploring More When It Needs in Deep Reinforcement Learning
• Learning to Explore by Reinforcement over High-Level Options
• How do Offline Measures for Exploration in Reinforcement Learning behave?
• Perturbation-based exploration methods in deep reinforcement learning
• ACDER: Augmented Curiosity-Driven Experience Replay
• State Distribution-aware Sampling for Deep Q-learning
• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy
• Leveraging the Variance of Return Sequences for Exploration Policy
• Combine PPO with NES to Improve Exploration
• Policy Manifold Search for Improving Diversity-based Neuroevolution