Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning

Abstract

This paper introduces Greedy UnMix (GUM) for cooperative multi-agent reinforcement learning (MARL). Greedy UnMix aims to avoid scenarios where MARL methods fail due to overestimation of values as part of the large joint state-action space. It aims to address this through a conservative Q-learning approach through restricting the state-marginal in the dataset to avoid unobserved joint state action spaces, whilst concurrently attempting to unmix or simplify the problem space under the centralized training with decentralized execution paradigm. We demonstrate the adherence to Q-function lower bounds in the Q-learning for MARL scenarios, and demonstrate superior performance to existing Q-learning MARL approaches as well as more general MARL algorithms over a set of benchmark MARL tasks, despite its relative simplicity compared with state-of-the-art approaches.

References in corpus (9)

• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Distributional Reinforcement Learning with Quantile Regression
• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• Safe Policy Improvement with Baseline Bootstrapping
• Safe Policy Improvement by Minimizing Robust Baseline Regret