Exploration by Random Network Distillation

Abstract

We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method to flexibly combine intrinsic and extrinsic rewards. We find that the random network distillation (RND) bonus combined with this increased flexibility enables significant progress on several hard exploration Atari games. In particular we establish state of the art performance on Montezuma's Revenge, a game famously difficult for deep reinforcement learning methods. To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level.

References in corpus (26)

• Playing Atari with Deep Reinforcement Learning
• Neural Architecture Search with Reinforcement Learning
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Noisy Networks for Exploration
• Distributed Prioritized Experience Replay
• Deep Exploration via Bootstrapped DQN
• Parameter Space Noise for Exploration
• Large-Scale Study of Curiosity-Driven Learning
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Unifying Count-Based Exploration and Intrinsic Motivation
• A Distributional Perspective on Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Randomized Prior Functions for Deep Reinforcement Learning
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Observe and Look Further: Achieving Consistent Performance on Atari
• UCB Exploration via Q-Ensembles
• Variational Option Discovery Algorithms
• The Uncertainty Bellman Equation and Exploration
• Learning to Perform Physics Experiments via Deep Reinforcement Learning
• DORA The Explorer: Directed Outreaching Reinforcement Action-Selection
• Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems
• Expert-augmented actor-critic for ViZDoom and Montezumas Revenge

Cited by in corpus (130)

• Dota 2 with Large Scale Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• Conservative Q-Learning for Offline Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Go-Explore: a New Approach for Hard-Exploration Problems
• Episodic Curiosity through Reachability
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• A survey on intrinsic motivation in reinforcement learning
• Efficient Exploration via State Marginal Matching
• Never Give Up: Learning Directed Exploration Strategies
• Planning to Explore via Self-Supervised World Models
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Hyperbolic Discounting and Learning over Multiple Horizons
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
• Dealing with Sparse Rewards in Reinforcement Learning
• Reinforcement Learning with Prototypical Representations
• Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• On Joint Learning for Solving Placement and Routing in Chip Design
• On Reward-Free Reinforcement Learning with Linear Function Approximation
• Information Theoretic Regret Bounds for Online Nonlinear Control
• The Ingredients of Real-World Robotic Reinforcement Learning
• A Survey of Embodied AI: From Simulators to Research Tasks
• Action and Perception as Divergence Minimization
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Approximate information state for approximate planning and reinforcement learning in partially observed systems
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Meta-learning curiosity algorithms
• RoboNet: Large-Scale Multi-Robot Learning
• Model-based Adversarial Meta-Reinforcement Learning
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Worst-Case Regret Bounds for Exploration via Randomized Value Functions
• Skynet: A Top Deep RL Agent in the Inaugural Pommerman Team Competition
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Reinforcement Learning with A* and a Deep Heuristic
• Optimistic Exploration even with a Pessimistic Initialisation
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• Energy-Based Imitation Learning
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Show Us the Way: Learning to Manage Dialog from Demonstrations
• Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
• DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
• Exploration and preference satisfaction trade-off in reward-free learning
• Evaluating Agents without Rewards
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Never Forget: Balancing Exploration and Exploitation via Learning Optical Flow
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Geometric Entropic Exploration
• Spatial Graph Attention and Curiosity-driven Policy for Antiviral Drug Discovery
• Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning
• Approximate Exploration through State Abstraction
• Exploring Exploration: Comparing Children with RL Agents in Unified Environments
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Temporal Difference Uncertainties as a Signal for Exploration
• Novelty Search in Representational Space for Sample Efficient Exploration
• Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation
• Learning Markov State Abstractions for Deep Reinforcement Learning
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning
• Population-Guided Parallel Policy Search for Reinforcement Learning
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• A Survey of Exploration Methods in Reinforcement Learning
• A Max-Min Entropy Framework for Reinforcement Learning
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• Influence-Based Multi-Agent Exploration
• SAFARI: Safe and Active Robot Imitation Learning with Imagination
• Implicit Generative Modeling for Efficient Exploration
• Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning
• On Catastrophic Interference in Atari 2600 Games
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
• Towards Automatic Actor-Critic Solutions to Continuous Control
• Learning Abstract Models for Strategic Exploration and Fast Reward Transfer
• Online reinforcement learning with sparse rewards through an active inference capsule
• Disentangling causal effects for hierarchical reinforcement learning
• Uncertainty-Aware Self-Supervised Target-Mass Grasping of Granular Foods
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Guided Exploration with Proximal Policy Optimization using a Single Demonstration
• Bridging In- and Out-of-distribution Samples for Their Better Discriminability
• Temporally-Extended ε-Greedy Exploration
• Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
• Novel Policy Seeking with Constrained Optimization
• Learning to Play by Imitating Humans
• Amplifying the Imitation Effect for Reinforcement Learning of UCAV's Mission Execution
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• Online Safety Assurance for Deep Reinforcement Learning
• Improving Model-Based Control and Active Exploration with Reconstruction Uncertainty Optimization
• DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Analyzing Visual Representations in Embodied Navigation Tasks
• On the Sensory Commutativity of Action Sequences for Embodied Agents
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Proposal Towards a Personalized Knowledge-powered Self-play Based Ensemble Dialog System
• ISL: A novel approach for deep exploration
• Intrinsic motivations and open-ended learning
• Benchmarking the Spectrum of Agent Capabilities
• Random Network Distillation as a Diversity Metric for Both Image and Text Generation
• Support-weighted Adversarial Imitation Learning
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• TAMPC: A Controller for Escaping Traps in Novel Environments
• Learning to Stop: Dynamic Simulation Monte-Carlo Tree Search
• A brain basis of dynamical intelligence for AI and computational neuroscience
• Maximum Entropy Model-based Reinforcement Learning
• When should agents explore?
• Explore and Control with Adversarial Surprise
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning
• Learning to Shape Rewards using a Game of Two Partners
• What classifiers know what they don't?
• Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments
• Importance of Environment Design in Reinforcement Learning: A Study of a Robotic Environment
• Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning
• Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems
• Optimising Stochastic Routing for Taxi Fleets with Model Enhanced Reinforcement Learning
• Imitation with Neural Density Models
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning
• ACDER: Augmented Curiosity-Driven Experience Replay
• Adversarial Machine Learning in Text Analysis and Generation
• Uncertainty Surrogates for Deep Learning
• A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals
• Evaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy
• Deep Reinforcement Learning for Complex Manipulation Tasks with Sparse Feedback
• Interpretable Few-Shot Learning via Linear Distillation
• MIME: Mutual Information Minimisation Exploration
• On the potential for open-endedness in neural networks
• Fixed $\beta$-VAE Encoding for Curious Exploration in Complex 3D Environments