A Survey and Critique of Multiagent Deep Reinforcement Learning

Abstract

Deep reinforcement learning (RL) has achieved outstanding results in recent years. This has led to a dramatic increase in the number of applications and methods. Recent works have explored learning beyond single-agent scenarios and have considered multiagent learning (MAL) scenarios. Initial results report successes in complex multiagent domains, although there are several challenges to be addressed. The primary goal of this article is to provide a clear overview of current multiagent deep reinforcement learning (MDRL) literature. Additionally, we complement the overview with a broader analysis: (i) we revisit previous key components, originally presented in MAL and RL, and highlight how they have been adapted to multiagent deep reinforcement learning settings. (ii) We provide general guidelines to new practitioners in the area: describing lessons learned from MDRL works, pointing to recent benchmarks, and outlining open avenues of research. (iii) We take a more critical tone raising practical challenges of MDRL (e.g., implementation and computational demands). We expect this article will help unify and motivate future research to take advantage of the abundant literature that exists (e.g., RL and MAL) in a joint effort to promote fruitful research in the multiagent community.

Under review since Oct 2018. Earlier versions of this work had the title: "Is multiagent deep reinforcement learning the answer or the question? A brief survey"

References in corpus (116)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning in Neural Networks: An Overview
• Distilling the Knowledge in a Neural Network
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Continuous control with deep reinforcement learning
• LSTM: A Search Space Odyssey
• Playing Atari with Deep Reinforcement Learning
• On the difficulty of training Recurrent Neural Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• Asynchronous Methods for Deep Reinforcement Learning
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• An Introduction to Deep Reinforcement Learning
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Deep Recurrent Q-Learning for Partially Observable MDPs
• StarCraft II: A New Challenge for Reinforcement Learning
• Emergence of Locomotion Behaviours in Rich Environments
• OpenAI Gym
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Unity: A General Platform for Intelligent Agents
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Value-Function Approximations for Partially Observable Markov Decision Processes
• Deep Reinforcement Learning: An Overview
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Robust Adversarial Reinforcement Learning
• Evolutionary Algorithms for Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Hindsight Experience Replay
• Mean Field Multi-Agent Reinforcement Learning
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• Learning Multiagent Communication with Backpropagation
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Optimal and Approximate Q-value Functions for Decentralized POMDPs
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• FeUdal Networks for Hierarchical Reinforcement Learning
• Population Based Training of Neural Networks
• Unifying Count-Based Exploration and Intrinsic Motivation
• The Hanabi Challenge: A New Frontier for AI Research
• Sample Efficient Actor-Critic with Experience Replay
• Go-Explore: a New Approach for Hard-Exploration Problems
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Learning Finite-State Controllers for Partially Observable Environments
• Equivalence Between Policy Gradients and Soft Q-Learning
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Efficient Solution Algorithms for Factored MDPs
• Emergent Complexity via Multi-Agent Competition
• A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning
• The Mechanics of n-Player Differentiable Games
• Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
• Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
• Multi-Agent Cooperation and the Emergence of (Natural) Language
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Opponent Modeling in Deep Reinforcement Learning
• RUDDER: Return Decomposition for Delayed Rewards
• Deep Reinforcement Learning and the Deadly Triad
• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• Accelerated Methods for Deep Reinforcement Learning
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• On the Origin of Deep Learning
• Policy Distillation
• Emergent Coordination Through Competition
• Learning Policy Representations in Multiagent Systems
• Actor-Critic Policy Optimization in Partially Observable Multiagent Environments
• Machine Theory of Mind
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Towards Characterizing Divergence in Deep Q-Learning
• Graying the black box: Understanding DQNs
• Multiagent Soft Q-Learning
• The Multi-Agent Reinforcement Learning in MalmÖ (MARLÖ) Competition
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• A Deep Policy Inference Q-Network for Multi-Agent Systems
• Deterministic Implementations for Reproducibility in Deep Reinforcement Learning
• Beating the World's Best at Super Smash Bros. with Deep Reinforcement Learning
• Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU
• A Geometric Perspective on Optimal Representations for Reinforcement Learning
• The Scientific Method in the Science of Machine Learning
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• A Comparative Analysis of Expected and Distributional Reinforcement Learning
• Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• Action-depedent Control Variates for Policy Optimization via Stein's Identity
• Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments
• Playing Atari with Six Neurons
• Agent Modeling as Auxiliary Task for Deep Reinforcement Learning
• The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors
• Skynet: A Top Deep RL Agent in the Inaugural Pommerman Team Competition
• Surprising Negative Results for Generative Adversarial Tree Search
• Microscopic Traffic Simulation by Cooperative Multi-agent Deep Reinforcement Learning
• Neural Replicator Dynamics
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Using Monte Carlo Tree Search as a Demonstrator within Asynchronous Deep RL
• Weighted Double Deep Multiagent Reinforcement Learning in Stochastic Cooperative Environments
• Convergence of Multi-Agent Learning with a Finite Step Size in General-Sum Games
• On Hard Exploration for Reinforcement Learning: a Case Study in Pommerman
• Towards Efficient Detection and Optimal Response against Sophisticated Opponents
• Modeling Friends and Foes

Cited by in corpus (91)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Social physics
• The Hanabi Challenge: A New Frontier for AI Research
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligence
• More Than Privacy: Applying Differential Privacy in Key Areas of Artificial Intelligence
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Game-Theoretic Multiagent Reinforcement Learning
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• Distributed Reinforcement Learning for Flexible and Efficient UAV Swarm Control
• Perspectives on adaptive dynamical systems
• Towards Joint Learning of Optimal MAC Signaling and Wireless Channel Access
• A DRL-based Multiagent Cooperative Control Framework for CAV Networks: a Graphic Convolution Q Network
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Coordination for Connected and Automated Vehicles at Non-signalized Intersections: A Value Decomposition-based Multiagent Deep Reinforcement Learning Approach
• "Other-Play" for Zero-Shot Coordination
• Safe Multi-Agent Reinforcement Learning via Shielding
• Boundary-aware Supervoxel-level Iteratively Refined Interactive 3D Image Segmentation with Multi-agent Reinforcement Learning
• Spatial Intention Maps for Multi-Agent Mobile Manipulation
• A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising
• Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention
• A Generalized Training Approach for Multiagent Learning
• Artificial Collective Intelligence Engineering: a Survey of Concepts and Perspectives
• Collaborative Visual Navigation
• Deep reinforcement learning of event-triggered communication and consensus-based control for distributed cooperative transport
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
• Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environments
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Skynet: A Top Deep RL Agent in the Inaugural Pommerman Team Competition
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Learning Multi-Agent Coordination through Connectivity-driven Communication
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• On Multi-Agent Learning in Team Sports Games
• Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning
• Difference Rewards Policy Gradients
• Algorithms in Multi-Agent Systems: A Holistic Perspective from Reinforcement Learning and Game Theory
• Real-time tree search with pessimistic scenarios
• Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Winning Isn't Everything: Enhancing Game Development with Intelligent Agents
• Cooperative Control of Mobile Robots with Stackelberg Learning
• Multi-Agent Deep Reinforcement Learning for Request Dispatching in Distributed-Controller Software-Defined Networking
• Emergent Language: A Survey and Taxonomy
• GHQ: Grouped Hybrid Q Learning for Heterogeneous Cooperative Multi-agent Reinforcement Learning
• The graph structure of two-player games
• Decentralized Q-Learning in Zero-sum Markov Games
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Emergent Road Rules In Multi-Agent Driving Environments
• Using Monte Carlo Tree Search as a Demonstrator within Asynchronous Deep RL
• TVDO: Tchebycheff Value-Decomposition Optimization for Multi-Agent Reinforcement Learning
• An Efficient Transfer Learning Framework for Multiagent Reinforcement Learning
• Distributed Reinforcement Learning for Cooperative Multi-Robot Object Manipulation
• Calibration of Shared Equilibria in General Sum Partially Observable Markov Games
• Deep Reinforcement Learning for Multi-Agent Coordination
• Safer Deep RL with Shallow MCTS: A Case Study in Pommerman
• Software Engineering for Collective Cyber-Physical Ecosystems
• Joint Attention for Multi-Agent Coordination and Social Learning
• Regret Bounds for Decentralized Learning in Cooperative Multi-Agent Dynamical Systems
• Coordination in Adversarial Sequential Team Games via Multi-Agent Deep Reinforcement Learning
• Attentional Policies for Cross-Context Multi-Agent Reinforcement Learning
• Reinforcement Learning for Intelligent Healthcare Systems: A Comprehensive Survey
• Multi-Agent Coordination in Adversarial Environments through Signal Mediated Strategies
• Resource allocation in dynamic multiagent systems
• Multi-Task Federated Reinforcement Learning with Adversaries
• Inter-Level Cooperation in Hierarchical Reinforcement Learning
• Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning
• A survey of air combat behavior modeling using machine learning
• Competitive Policy Optimization
• Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents
• Deep Q-Network Based Multi-agent Reinforcement Learning with Binary Action Agents
• Adaptive Online Distributed Optimal Control of Very-Large-Scale Robotic Systems
• Decentralized Reinforcement Learning for Multi-Target Search and Detection by a Team of Drones
• Emergent cooperation through mutual information maximization
• Reinforcement Learning In Two Player Zero Sum Simultaneous Action Games
• Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning
• Back to Basics: Deep Reinforcement Learning in Traffic Signal Control
• Multi-Agent Reinforcement Learning as a Computational Tool for Language Evolution Research: Historical Context and Future Challenges
• DM${}^2$: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching
• Multi-Agent Intention Sharing via Leader-Follower Forest
• Inducing Cooperation via Team Regret Minimization based Multi-Agent Deep Reinforcement Learning
• Decentralized Circle Formation Control for Fish-like Robots in the Real-world via Reinforcement Learning
• Network Distributed Multi-Agent Reinforcement Learning for Consensus Control of Quadcopters
• A new soft computing method for integration of expert's knowledge in reinforcement learn-ing problems
• Scalable, Decentralized Multi-Agent Reinforcement Learning Methods Inspired by Stigmergy and Ant Colonies
• The Power of Communication in a Distributed Multi-Agent System
• SA-MATD3:Self-attention-based multi-agent continuous control method in cooperative environments
• Multi-agent Reinforcement Learning in OpenSpiel: A Reproduction Report
• Improved Cooperation by Exploiting a Common Signal