Unifying Count-Based Exploration and Intrinsic Motivation

Abstract

We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use density models to measure uncertainty, and propose a novel algorithm for deriving a pseudo-count from an arbitrary density model. This technique enables us to generalize count-based exploration algorithms to the non-tabular case. We apply our ideas to Atari 2600 games, providing sensible pseudo-counts from raw pixels. We transform these pseudo-counts into intrinsic rewards and obtain significantly improved exploration in a number of hard games, including the infamously difficult Montezuma's Revenge.

References in corpus (6)

• Asynchronous Methods for Deep Reinforcement Learning
• Pixel Recurrent Neural Networks
• VIME: Variational Information Maximizing Exploration
• Thompson Sampling is Asymptotically Optimal in General Environments
• Domain-Independent Optimistic Initialization for Reinforcement Learning
• Laplace's rule of succession in information geometry

Cited by in corpus (68)

• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Large-Scale Study of Curiosity-Driven Learning
• Exploration by Random Network Distillation
• First return, then explore
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Automatic Goal Generation for Reinforcement Learning Agents
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• Investigating Human Priors for Playing Video Games
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• Evolutionary Reinforcement Learning: A Survey
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes
• Contingency-Aware Exploration in Reinforcement Learning
• The Uncertainty Bellman Equation and Exploration
• Unsupervised Meta-Learning for Reinforcement Learning
• ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Beating Atari with Natural Language Guided Reinforcement Learning
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• Computational Theories of Curiosity-Driven Learning
• AdaptiveFL: Adaptive Heterogeneous Federated Learning for Resource-Constrained AIoT Systems
• Learning to Explore with Meta-Policy Gradient
• Interactive Fiction Games: A Colossal Adventure
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Multi-agent Deep Reinforcement Learning with Extremely Noisy Observations
• Autonomous development and learning in artificial intelligence and robotics: Scaling up deep learning to human--like learning
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Variational Bayesian Reinforcement Learning with Regret Bounds
• Leveraging exploration in off-policy algorithms via normalizing flows
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Smooth Exploration for Robotic Reinforcement Learning
• Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning
• Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks
• Deep Abstract Q-Networks
• Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning
• Expert-augmented actor-critic for ViZDoom and Montezumas Revenge
• Taking the Scenic Route: Automatic Exploration for Videogames
• Challenges of Context and Time in Reinforcement Learning: Introducing Space Fortress as a Benchmark
• Generating Automatic Curricula via Self-Supervised Active Domain Randomization
• Flow-based Intrinsic Curiosity Module
• Hashing over Predicted Future Frames for Informed Exploration of Deep Reinforcement Learning
• Exploration Bonus for Regret Minimization in Undiscounted Discrete and Continuous Markov Decision Processes
• AutoPhase: Compiler Phase-Ordering for High Level Synthesis with Deep Reinforcement Learning
• Identifying Critical States by the Action-Based Variance of Expected Return
• Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning
• On Catastrophic Interference in Atari 2600 Games
• Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards
• Switching Isotropic and Directional Exploration with Parameter Space Noise in Deep Reinforcement Learning
• Improved Exploring Starts by Kernel Density Estimation-Based State-Space Coverage Acceleration in Reinforcement Learning
• Towards Robust Bisimulation Metric Learning
• Exploration of Self-Propelling Droplets Using a Curiosity Driven Robotic Assistant
• Self-Paced Deep Reinforcement Learning
• Solving Atari Games Using Fractals And Entropy
• Sidekick Policy Learning for Active Visual Exploration
• Reusability and Transferability of Macro Actions for Reinforcement Learning
• A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning
• Perturbation-based exploration methods in deep reinforcement learning
• Shared Learning : Enhancing Reinforcement in $Q$-Ensembles
• Neural Embedding for Physical Manipulations
• Dex: Incremental Learning for Complex Environments in Deep Reinforcement Learning
• Region Growing Curriculum Generation for Reinforcement Learning