Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
arXiv:2011.09533
Abstract
Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function. In this paper, we demonstrate that, despite its various theoretical shortcomings, Independent PPO (IPPO), a form of independent learning in which each agent simply estimates its local value function, can perform just as well as or better than state-of-the-art joint learning approaches on popular multi-agent benchmark suite SMAC with little hyperparameter tuning. We also compare IPPO to several variants; the results suggest that IPPO's strong performance may be due to its robustness to some forms of environment non-stationarity.
References in corpus (12)
- Asynchronous Methods for Deep Reinforcement Learning
- StarCraft II: A New Challenge for Reinforcement Learning
- Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
- Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
- FACMAC: Factored Multi-Agent Centralised Policy Gradients
- Guided Deep Reinforcement Learning for Swarm Systems
- MAVEN: Multi-Agent Variational Exploration
- Deep Coordination Graphs
- Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
- Revisiting Design Choices in Proximal Policy Optimization
Cited by in corpus (9)
- Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
- MADRL-Based Rate Adaptation for 360° Video Streaming with Multi-Viewpoint Prediction
- Settling the Variance of Multi-Agent Policy Gradients
- Collaborative Visual Navigation
- Multi-Agent Constrained Policy Optimisation
- Strategic Decision-Making in Multi-Agent Domains: A Weighted Constrained Potential Dynamic Game Approach
- Stimulate the Potential of Robots via Competition
- Joint Attention for Multi-Agent Coordination and Social Learning
- Reinforcement Learning with Success Induced Task Prioritization