A survey on intrinsic motivation in reinforcement learning

Abstract

The reinforcement learning (RL) research area is very active, with an important number of new contributions; especially considering the emergent field of deep RL (DRL). However a number of scientific and technical challenges still need to be addressed, amongst which we can mention the ability to abstract actions or the difficulty to explore the environment which can be addressed by intrinsic motivation (IM). In this article, we provide a survey on the role of intrinsic motivation in DRL. We categorize the different kinds of intrinsic motivations and detail for each category, its advantages and limitations with respect to the mentioned challenges. Additionnally, we conduct an in-depth investigation of substantial current research questions, that are currently under study or not addressed at all in the considered research area of DRL. We choose to survey these research works, from the perspective of learning how to achieve tasks. We suggest then, that solving current challenges could lead to a larger developmental architecture which may tackle most of the tasks. We describe this developmental architecture on the basis of several building blocks composed of a RL algorithm and an IM module compressing information.

References in corpus (73)

• Continuous control with deep reinforcement learning
• Overcoming catastrophic forgetting in neural networks
• The information bottleneck method
• Prioritized Experience Replay
• Conditional Image Generation with PixelCNN Decoders
• Weight Uncertainty in Neural Networks
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Noisy Networks for Exploration
• VIME: Variational Information Maximizing Exploration
• Parameter Space Noise for Exploration
• Deep Variational Information Bottleneck
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Visual Reinforcement Learning with Imagined Goals
• State Representation Learning for Control: An Overview
• Deep Q-learning from Demonstrations
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Data-Efficient Hierarchical Reinforcement Learning
• Exploration by Random Network Distillation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Episodic Curiosity through Reachability
• Variational Intrinsic Control
• Learning and Transfer of Modulated Locomotor Controllers
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Efficient Exploration via State Marginal Matching
• Deep Successor Reinforcement Learning
• Model-Based Active Exploration
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Variational Option Discovery Algorithms
• Dynamics-Aware Unsupervised Discovery of Skills
• A multi-agent reinforcement learning model of common-pool resource appropriation
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Independently Controllable Factors
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• Curiosity-Driven Experience Prioritization via Density Estimation
• Hierarchical Reinforcement Learning with Hindsight
• Learning Abstract Options
• Independently Controllable Features
• DORA The Explorer: Directed Outreaching Reinforcement Action-Selection
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Decoupling Dynamics and Reward for Transfer Learning
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• EMI: Exploration with Mutual Information
• Self-supervised Learning of Image Embedding for Continuous Control
• Boltzmann Exploration Done Right
• A unified strategy for implementing curiosity and empowerment driven reinforcement learning
• Reward Shaping with Recurrent Neural Networks for Speeding up On-Line Policy Learning in Spoken Dialogue Systems
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics
• Learning latent state representation for speeding up exploration
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
• Symmetry-Based Disentangled Representation Learning requires Interaction with Environments
• Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
• Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the World
• Rethinking Continual Learning for Autonomous Agents and Robots
• On the Complexity of Exploration in Goal-Driven Navigation
• The Journey is the Reward: Unsupervised Learning of Influential Trajectories
• Simple Algorithmic Principles of Discovery, Subjective Beauty, Selective Attention, Curiosity & Creativity

Cited by in corpus (19)

• Exploration in Deep Reinforcement Learning: A Survey
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Review: Deep Learning in Electron Microscopy
• Neurosymbolic Reinforcement Learning and Planning: A Survey
• Reinforcement learning for optimization of variational quantum circuit architectures
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Curiosity in exploring chemical space: Intrinsic rewards for deep molecular reinforcement learning
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning
• Deep KKL: Data-driven Output Prediction for Non-Linear Systems
• Reward Bonuses with Gain Scheduling Inspired by Iterative Deepening Search
• Active World Model Learning with Progress Curiosity
• Learning What to Memorize: Using Intrinsic Motivation to Form Useful Memory in Partially Observable Reinforcement Learning
• Complex Robotic Manipulation via Graph-Based Hindsight Goal Generation
• Curiosity creates Diversity in Policy Search
• Intrinsically motivated option learning: a comparative study of recent methods
• Towards Robust Bisimulation Metric Learning
• Exploring Unknown States with Action Balance
• Learning in Sparse Rewards settings through Quality-Diversity algorithms
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents