Settling the Variance of Multi-Agent Policy Gradients
arXiv:2108.08612
Abstract
Policy gradient (PG) methods are popular reinforcement learning (RL) methods where a baseline is often applied to reduce the variance of gradient estimates. In multi-agent RL (MARL), although the PG theorem can be naturally extended, the effectiveness of multi-agent PG (MAPG) methods degrades as the variance of gradient estimates increases rapidly with the number of agents. In this paper, we offer a rigorous analysis of MAPG methods by, firstly, quantifying the contributions of the number of agents and agents' explorations to the variance of MAPG estimators. Based on this analysis, we derive the optimal baseline (OB) that achieves the minimal variance. In comparison to the OB, we measure the excess variance of existing MARL algorithms such as vanilla MAPG and COMA. Considering using deep neural networks, we also propose a surrogate version of OB, which can be seamlessly plugged into any existing PG methods in MARL. On benchmarks of Multi-Agent MuJoCo and StarCraft challenges, our OB technique effectively stabilises training and improves the performance of multi-agent PPO and COMA algorithms by a significant margin.
References in corpus (18)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Trust Region Policy Optimization
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Counterfactual Multi-Agent Policy Gradients
- Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
- Mean Field Multi-Agent Reinforcement Learning
- Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
- QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
- Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
- The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
- Game-Theoretic Multiagent Reinforcement Learning
- FACMAC: Factored Multi-Agent Centralised Policy Gradients
- SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
- Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
- Off-Policy Multi-Agent Decomposed Policy Gradients
- Expected Policy Gradients for Reinforcement Learning
- Multi-Agent Determinantal Q-Learning