Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

Abstract

Deep Reinforcement Learning (DRL) and Deep Multi-agent Reinforcement Learning (MARL) have achieved significant successes across a wide range of domains, including game AI, autonomous vehicles, robotics, and so on. However, DRL and deep MARL agents are widely known to be sample inefficient that millions of interactions are usually needed even for relatively simple problem settings, thus preventing the wide application and deployment in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how efficiently exploring the environment and collecting informative experiences that could benefit policy learning towards the optimal ones. This problem becomes more challenging in complex environments with sparse rewards, noisy distractions, long horizons, and non-stationary co-learners. In this paper, we conduct a comprehensive survey on existing exploration methods for both single-agent and multi-agent RL. We start the survey by identifying several key challenges to efficient exploration. Beyond the above two main branches, we also include other notable exploration methods with different ideas and techniques. In addition to algorithmic analysis, we provide a comprehensive and unified empirical comparison of different exploration methods for DRL on a set of commonly used benchmarks. According to our algorithmic and empirical investigation, we finally summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.

arXiv commentAccepted by IEEE Transactions on Neural Networks and Learning Systems (TNNLS)

References in corpus (75)

• Continuous control with deep reinforcement learning
• A Brief Survey of Deep Reinforcement Learning
• Trust Region Policy Optimization
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Noisy Networks for Exploration
• Exploration in Deep Reinforcement Learning: A Survey
• VIME: Variational Information Maximizing Exploration
• Distributed Prioritized Experience Replay
• Deep Exploration via Bootstrapped DQN
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Parameter Space Noise for Exploration
• Large-Scale Study of Curiosity-Driven Learning
• Count-Based Exploration with Neural Density Models
• Visual Reinforcement Learning with Imagined Goals
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Safe Exploration in Continuous Action Spaces
• Reward Constrained Policy Optimization
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• (More) Efficient Reinforcement Learning via Posterior Sampling
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Episodic Curiosity through Reachability
• Successor Features for Transfer in Reinforcement Learning
• Variational Intrinsic Control
• Deep Exploration via Randomized Value Functions
• Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space
• A Multi-Objective Deep Reinforcement Learning Framework
• Self-Imitation Learning
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• Safe Exploration in Finite Markov Decision Processes with Gaussian Processes
• Randomized Prior Functions for Deep Reinforcement Learning
• A survey on intrinsic motivation in reinforcement learning
• Efficient Exploration via State Marginal Matching
• Generalization and Exploration via Randomized Value Functions
• Learning Montezuma's Revenge from a Single Demonstration
• Variational Option Discovery Algorithms
• Information Directed Sampling and Bandits with Heteroscedastic Noise
• Bootstrapped Thompson Sampling and Deep Exploration
• Contingency-Aware Exploration in Reinforcement Learning
• The Uncertainty Bellman Equation and Exploration
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Maximum Entropy-Regularized Multi-Goal Reinforcement Learning
• Reinforcement Learning with Prototypical Representations
• Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration
• DORA The Explorer: Directed Outreaching Reinforcement Action-Selection
• Hindsight policy gradients
• Cooperative Exploration for Multi-Agent Deep Reinforcement Learning
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• Conservative Safety Critics for Exploration
• A New Framework for Multi-Agent Reinforcement Learning -- Centralized Training and Exploration with Decentralized Execution via Policy Distillation
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• Regret Analysis of the Anytime Optimally Confident UCB Algorithm
• Learning latent state representation for speeding up exploration
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
• Scalable Coordinated Exploration in Concurrent Reinforcement Learning
• Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Towards Safe Reinforcement Learning with a Safety Editor Policy
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• Directed Exploration for Reinforcement Learning
• MULEX: Disentangling Exploitation from Exploration in Deep RL
• Ready Policy One: World Building Through Active Learning
• Self-Imitation Advantage Learning
• Principled Exploration via Optimistic Bootstrapping and Backward Induction
• Co-Imitation Learning without Expert Demonstration
• Efficient exploration of zero-sum stochastic games

Cited by in corpus (3)

• GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
• Inverse RL Scene Dynamics Learning for Nonlinear Predictive Control in Autonomous Vehicles
• Meta-Offline and Distributional Multi-Agent RL for Risk-Aware Decision-Making