Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

Abstract

Exploration in complex domains is a key challenge in reinforcement learning, especially for tasks with very sparse rewards. Recent successes in deep reinforcement learning have been achieved mostly using simple heuristic exploration strategies such as $\epsilon$-greedy action selection or Gaussian control noise, but there are many tasks where these methods are insufficient to make any learning progress. Here, we consider more complex heuristics: efficient and scalable exploration strategies that maximize a notion of an agent's surprise about its experiences via intrinsic motivation. We propose to learn a model of the MDP transition probabilities concurrently with the policy, and to form intrinsic rewards that approximate the KL-divergence of the true transition probabilities from the learned model. One of our approximations results in using surprisal as intrinsic motivation, while the other gives the $k$-step learning progress. We show that our incentives enable agents to succeed in a wide range of environments with high-dimensional state spaces and very sparse rewards, including continuous control tasks and games in the Atari RAM domain, outperforming several other heuristic exploration techniques.

Appeared in Deep RL Workshop at NIPS 2016

References in corpus (6)

• Dueling Network Architectures for Deep Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• VIME: Variational Information Maximizing Exploration
• Massively Parallel Methods for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic Environments

Cited by in corpus (45)

• Emergent Tool Use From Multi-Agent Autocurricula
• Parameter Space Noise for Exploration
• Large-Scale Study of Curiosity-Driven Learning
• Exploration by Random Network Distillation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Automatic Goal Generation for Reinforcement Learning Agents
• A survey on intrinsic motivation in reinforcement learning
• Intrinsically motivated reinforcement learning for human-robot interaction in the real-world
• Planning to Explore via Self-Supervised World Models
• Contingency-Aware Exploration in Reinforcement Learning
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• See, Hear, Explore: Curiosity via Audio-Visual Association
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Self-Supervised Exploration via Disagreement
• World Discovery Models
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Learning-Driven Exploration for Reinforcement Learning
• Autonomous exploration for navigating in non-stationary CMPs
• Exploratory State Representation Learning
• Novelty Search in Representational Space for Sample Efficient Exploration
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Bayesian Curiosity for Efficient Exploration in Reinforcement Learning
• Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation
• Active World Model Learning with Progress Curiosity
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• A Max-Min Entropy Framework for Reinforcement Learning
• Deep Reinforcement Learning in Fluid Mechanics: a promising method for both Active Flow Control and Shape Optimization
• Hashing over Predicted Future Frames for Informed Exploration of Deep Reinforcement Learning
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• Efficient Exploration through Intrinsic Motivation Learning for Unsupervised Subgoal Discovery in Model-Free Hierarchical Reinforcement Learning
• Towards Robust Bisimulation Metric Learning
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• State-Aware Variational Thompson Sampling for Deep Q-Networks
• Explore and Control with Adversarial Surprise
• MIME: Mutual Information Minimisation Exploration
• Being curious about the answers to questions: novelty search with learned attention
• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy
• Fixed $\beta$-VAE Encoding for Curious Exploration in Complex 3D Environments
• Neural Embedding for Physical Manipulations