Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning

Abstract

Recent developments in deep reinforcement learning are concerned with creating decision-making agents which can perform well in various complex domains. A particular approach which has received increasing attention is multi-agent reinforcement learning, in which multiple agents learn concurrently to coordinate their actions. In such multi-agent environments, additional learning problems arise due to the continually changing decision-making policies of agents. This paper surveys recent works that address the non-stationarity problem in multi-agent deep reinforcement learning. The surveyed methods range from modifications in the training procedure, such as centralized training, to learning representations of the opponent's policy, meta-learning, communication, and decentralized learning. The survey concludes with a list of open problems and possible lines of future research.

References in corpus (6)

• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks

Cited by in corpus (24)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control
• Intelligent Electric Vehicle Charging Recommendation Based on Multi-Agent Reinforcement Learning
• Stabilizing Generative Adversarial Networks: A Survey
• Beyond Robustness: A Taxonomy of Approaches towards Resilient Multi-Robot Systems
• Robust Multi-agent Communication via Multi-view Message Certification
• Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies
• Agent-Based Modelling for Urban Analytics: State of the Art and Challenges
• Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning
• Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing
• Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environments
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning
• Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• Learning to Robustly Negotiate Bi-Directional Lane Usage in High-Conflict Driving Scenarios
• Variational Autoencoders for Opponent Modeling in Multi-Agent Systems
• Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Indoor Path Planning for Multiple Unmanned Aerial Vehicles via Curriculum Learning
• Multi-Agent Deep Reinforcement Learning with Adaptive Policies
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems
• Coordinated Proximal Policy Optimization