An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey

Abstract

The reinforcement learning (RL) research area is very active, with an important number of new contributions; especially considering the emergent field of deep RL (DRL). However a number of scientific and technical challenges still need to be resolved, amongst which we can mention the ability to abstract actions or the difficulty to explore the environment in sparse-reward settings which can be addressed by intrinsic motivation (IM). We propose to survey these research works through a new taxonomy based on information theory: we computationally revisit the notions of surprise, novelty and skill learning. This allows us to identify advantages and disadvantages of methods and exhibit current outlooks of research. Our analysis suggests that novelty and surprise can assist the building of a hierarchy of transferable skills that further abstracts the environment and makes the exploration process more robust.

References in corpus (43)

• A Simple Framework for Contrastive Learning of Visual Representations
• Estimating Mutual Information
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Bootstrap your own latent: A new approach to self-supervised Learning
• Spectral Normalization for Generative Adversarial Networks
• Learning deep representations by mutual information estimation and maximization
• An Introduction to Deep Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• First return, then explore
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• Diverse Trajectory Forecasting with Determinantal Point Processes
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Never Give Up: Learning Directed Exploration Strategies
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Planning to Explore via Self-Supervised World Models
• A non-parametric k-nearest neighbour entropy estimator
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Maximum Entropy-Regularized Multi-Goal Reinforcement Learning
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• EMI: Exploration with Mutual Information
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Fast Task Inference with Variational Intrinsic Successor Features
• Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
• Efficient Intrinsically Motivated Robotic Grasping with Learning-Adaptive Imagination in Latent Space
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Latent World Models For Intrinsically Motivated Exploration
• Sub-policy Adaptation for Hierarchical Reinforcement Learning
• Novelty Search in Representational Space for Sample Efficient Exploration
• Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
• Unsupervised Skill Discovery with Bottleneck Option Learning