DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
arXiv:2102.07936
Abstract
In fully cooperative multi-agent reinforcement learning (MARL) settings, the environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of the other agents. To address the above issues, we integrate distributional RL and value function factorization methods by proposing a Distributional Value Function Factorization (DFAC) framework to generalize expected value function factorization methods to their DFAC variants. DFAC extends the individual utility functions from deterministic variables to random variables, and models the quantile function of the total return as a quantile mixture. To validate DFAC, we demonstrate DFAC's ability to factorize a simple two-step matrix game with stochastic rewards and perform experiments on all Super Hard tasks of StarCraft Multi-Agent Challenge, showing that DFAC is able to outperform expected value function factorization baselines.
arXiv commentICML 2021
References in corpus (12)
- Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Implicit Quantile Networks for Distributional Reinforcement Learning
- QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
- Distributional Reinforcement Learning with Quantile Regression
- Fully Parameterized Quantile Function for Distributional Reinforcement Learning
- ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
- Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control
- Learning Nearly Decomposable Value Functions Via Communication Minimization
- A Comparative Analysis of Expected and Distributional Reinforcement Learning
- Information-Directed Exploration for Deep Reinforcement Learning
- Distributional Reward Decomposition for Reinforcement Learning