Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
arXiv:2006.00587
Abstract
Value factorization is a popular and promising approach to scaling up multi-agent reinforcement learning in cooperative settings, which balances the learning scalability and the representational capacity of value functions. However, the theoretical understanding of such methods is limited. In this paper, we formalize a multi-agent fitted Q-iteration framework for analyzing factorized multi-agent Q-learning. Based on this framework, we investigate linear value factorization and reveal that multi-agent Q-learning with this simple decomposition implicitly realizes a powerful counterfactual credit assignment, but may not converge in some settings. Through further analysis, we find that on-policy training or richer joint value function classes can improve its local or global convergence properties, respectively. Finally, to support our theoretical implications in practical realization, we conduct an empirical analysis of state-of-the-art deep multi-agent Q-learning algorithms on didactic examples and a broad set of StarCraft II unit micromanagement tasks.
arXiv commentThirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021)
References in corpus (20)
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Counterfactual Multi-Agent Policy Gradients
- Emergent Tool Use From Multi-Agent Autocurricula
- QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
- Learning Finite-State Controllers for Partially Observable Environments
- Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
- QPLEX: Duplex Dueling Multi-Agent Q-Learning
- FACMAC: Factored Multi-Agent Centralised Policy Gradients
- Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
- Deep Reinforcement Learning and the Deadly Triad
- Guided Deep Reinforcement Learning for Swarm Systems
- Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
- Provably efficient RL with Rich Observations via Latent State Decoding
- Information-Theoretic Considerations in Batch Reinforcement Learning
- Near-optimal Reinforcement Learning in Factored MDPs
- Diagnosing Bottlenecks in Deep Q-learning Algorithms
- DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
- Reinforcement Learning in Rich-Observation MDPs using Spectral Methods
- Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning