QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
arXiv:2009.04197
Abstract
In Cooperative Multi-Agent Reinforcement Learning (MARL) and under the setting of Centralized Training with Decentralized Execution (CTDE), agents observe and interact with their environment locally and independently. With local observation and random sampling, the randomness in rewards and observations leads to randomness in long-term returns. Existing methods such as Value Decomposition Network (VDN) and QMIX estimate the value of long-term returns as a scalar that does not contain the information of randomness. Our proposed model QR-MIX introduces quantile regression, modeling joint state-action values as a distribution, combining QMIX with Implicit Quantile Network (IQN). However, the monotonicity in QMIX limits the expression of joint state-action value distribution and may lead to incorrect estimation results in non-monotonic cases. Therefore, we proposed a flexible loss function to approximate the monotonicity found in QMIX. Our model is not only more tolerant of the randomness of returns, but also more tolerant of the randomness of monotonic constraints. The experimental results demonstrate that QR-MIX outperforms the previous state-of-the-art method QMIX in the StarCraft Multi-Agent Challenge (SMAC) environment.
arXiv commentThere are some experimental errors and experimental unfairness in this paper that will seriously affect the later studies
References in corpus (17)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Deep Recurrent Q-Learning for Partially Observable MDPs
- A Survey and Critique of Multiagent Deep Reinforcement Learning
- Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Implicit Quantile Networks for Distributional Reinforcement Learning
- A Distributional Perspective on Reinforcement Learning
- QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
- Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
- FACMAC: Factored Multi-Agent Centralised Policy Gradients
- Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
- Fully Parameterized Quantile Function for Distributional Reinforcement Learning
- Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning