Large-Scale Study of Curiosity-Driven Learning

Abstract

Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal. In this paper: (a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance, and a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many game environments. (b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.). (c) We demonstrate limitations of the prediction-based rewards in stochastic setups. Game-play videos and code are at https://pathak22.github.io/large-scale-curiosity/

First three authors contributed equally and ordered alphabetically. Website at https://pathak22.github.io/large-scale-curiosity/

References in corpus (15)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Noisy Networks for Exploration
• VIME: Variational Information Maximizing Exploration
• Deep Exploration via Bootstrapped DQN
• Parameter Space Noise for Exploration
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Variational Intrinsic Control
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• UCB Exploration via Q-Ensembles
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Computational Theories of Curiosity-Driven Learning

Cited by in corpus (86)

• Exploration in Deep Reinforcement Learning: A Survey
• Exploration by Random Network Distillation
• Episodic Curiosity through Reachability
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Computer-inspired Quantum Experiments
• Never Give Up: Learning Directed Exploration Strategies
• Contingency-Aware Exploration in Reinforcement Learning
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Dealing with Sparse Rewards in Reinforcement Learning
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• Automated curricula through setter-solver interactions
• Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning
• A Survey of Embodied AI: From Simulators to Research Tasks
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• On Bonus-Based Exploration Methods in the Arcade Learning Environment
• Curiosity in exploring chemical space: Intrinsic rewards for deep molecular reinforcement learning
• Fast Task Inference with Variational Intrinsic Successor Features
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Model-based Adversarial Meta-Reinforcement Learning
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
• Learning to Reason in Large Theories without Imitation
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• World Discovery Models
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Optimistic Bull or Pessimistic Bear: Adaptive Deep Reinforcement Learning for Stock Portfolio Allocation
• Directed Exploration for Reinforcement Learning
• Never Forget: Balancing Exploration and Exploitation via Learning Optical Flow
• ELLA: Exploration through Learned Language Abstraction
• Novelty Search in Representational Space for Sample Efficient Exploration
• Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation
• Exploratory State Representation Learning
• Temporal Difference Uncertainties as a Signal for Exploration
• Taking the Scenic Route: Automatic Exploration for Videogames
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• Intrinsic Motivation for Encouraging Synergistic Behavior
• Influence-Based Multi-Agent Exploration
• Active World Model Learning with Progress Curiosity
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Generating Automatic Curricula via Self-Supervised Active Domain Randomization
• Multi-Agent Reinforcement Learning for Long-Term Network Resource Allocation through Auction: a V2X Application
• Implicit Generative Modeling for Efficient Exploration
• The MineRL BASALT Competition on Learning from Human Feedback
• A Quantum States Preparation Method Based on Difference-Driven Reinforcement Learning
• Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning
• Empowerment-driven Exploration using Mutual Information Estimation
• Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning
• Curiosity-Driven Recommendation Strategy for Adaptive Learning via Deep Reinforcement Learning
• Competitive Experience Replay
• Shaping embodied agent behavior with activity-context priors from egocentric video
• Mutual Information State Intrinsic Control
• Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Learning Latent State Spaces for Planning through Reward Prediction
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Amplifying the Imitation Effect for Reinforcement Learning of UCAV's Mission Execution
• Novel Policy Seeking with Constrained Optimization
• Policy Gradient from Demonstration and Curiosity
• Co-Imitation Learning without Expert Demonstration
• CLAIM: Curriculum Learning Policy for Influence Maximization in Unknown Social Networks
• Exploratory Test Agents for Stateful Software Systems
• Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning
• Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
• Learning the Arrow of Time
• Towards Robust Bisimulation Metric Learning
• Towards Tractable Optimism in Model-Based Reinforcement Learning
• Self-Supervised Contextual Bandits in Computer Vision
• ISL: A novel approach for deep exploration
• Benchmarking the Spectrum of Agent Capabilities
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Interaction Networks: Using a Reinforcement Learner to train other Machine Learning algorithms
• A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals
• Evaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks
• Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments
• Generative Exploration and Exploitation
• MIME: Mutual Information Minimisation Exploration
• ACDER: Augmented Curiosity-Driven Experience Replay
• Learning in Sparse Rewards settings through Quality-Diversity algorithms
• Reusability and Transferability of Macro Actions for Reinforcement Learning
• Lifetime policy reuse and the importance of task capacity