Exploration in Deep Reinforcement Learning: A Survey

Abstract

This paper reviews exploration techniques in deep reinforcement learning. Exploration techniques are of primary importance when solving sparse reward problems. In sparse reward problems, the reward is rare, which means that the agent will not find the reward often by acting randomly. In such a scenario, it is challenging for reinforcement learning to learn rewards and actions association. Thus more sophisticated exploration methods need to be devised. This review provides a comprehensive overview of existing exploration approaches, which are categorized based on the key contributions as follows reward novel states, reward diverse behaviours, goal-based methods, probabilistic methods, imitation-based methods, safe exploration and random-based methods. Then, the unsolved challenges are discussed to provide valuable future research directions. Finally, the approaches of different categories are compared in terms of complexity, computational effort and overall performance.

References in corpus (55)

• Continuous control with deep reinforcement learning
• A Brief Survey of Deep Reinforcement Learning
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Noisy Networks for Exploration
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Deep Exploration via Bootstrapped DQN
• Parameter Space Noise for Exploration
• Large-Scale Study of Curiosity-Driven Learning
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Count-Based Exploration with Neural Density Models
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Safe Exploration in Continuous Action Spaces
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Go-Explore: a New Approach for Hard-Exploration Problems
• Episodic Curiosity through Reachability
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Deep Exploration via Randomized Value Functions
• Self-Imitation Learning
• Reinforcement Learning in Healthcare: A Survey
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• Safe Exploration in Finite Markov Decision Processes with Gaussian Processes
• Randomized Prior Functions for Deep Reinforcement Learning
• A survey on intrinsic motivation in reinforcement learning
• Safe Exploration of State and Action Spaces in Reinforcement Learning
• Generalization and Exploration via Randomized Value Functions
• Learning Montezuma's Revenge from a Single Demonstration
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Information theoretic approach to interactive learning
• Eigenoption Discovery through the Deep Successor Representation
• Contingency-Aware Exploration in Reinforcement Learning
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• The Uncertainty Bellman Equation and Exploration
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Autonomous Quadrotor Landing using Deep Reinforcement Learning
• Curiosity-Driven Experience Prioritization via Density Estimation
• Fast active learning for pure exploration in reinforcement learning
• Why is Posterior Sampling Better than Optimism for Reinforcement Learning?
• Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning
• See, Hear, Explore: Curiosity via Audio-Visual Association
• Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards
• Curiosity-driven reinforcement learning with homeostatic regulation
• Learning-Driven Exploration for Reinforcement Learning
• Directed Exploration for Reinforcement Learning
• On Hard Exploration for Reinforcement Learning: a Case Study in Pommerman
• Learning Abstract Models for Strategic Exploration and Fast Reward Transfer
• Maximum Entropy Diverse Exploration: Disentangling Maximum Entropy Reinforcement Learning

Cited by in corpus (10)

• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape
• A social path to human-like artificial intelligence
• Advanced Persistent Threats (APT) Attribution Using Deep Reinforcement Learning
• Graph Exploration for Effective Multi-agent Q-Learning
• Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Sampling
• GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
• Whittle Index Based User Association in Dense Millimeter Wave Networks
• (Ir)rationality in AI: State of the Art, Research Challenges and Open Questions
• MIMIC: Integrating Diverse Personality Traits for Better Game Testing Using Large Language Model