Regularized Softmax Deep Multi-Agent $Q$-Learning

Abstract

Tackling overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning, but has received comparatively little attention in the multi-agent setting. In this work, we empirically demonstrate that QMIX, a popular $Q$-learning algorithm for cooperative multi-agent reinforcement learning (MARL), suffers from a more severe overestimation in practice than previously acknowledged, and is not mitigated by existing approaches. We rectify this with a novel regularization-based update scheme that penalizes large joint action-values that deviate from a baseline and demonstrate its effectiveness in stabilizing learning. Furthermore, we propose to employ a softmax operator, which we efficiently approximate in a novel way in the multi-agent setting, to further reduce the potential overestimation bias. Our approach, Regularized Softmax (RES) Deep Multi-Agent $Q$-Learning, is general and can be applied to any $Q$-learning based MARL algorithm. We demonstrate that, when applied to QMIX, RES avoids severe overestimation and significantly improves performance, yielding state-of-the-art results in a variety of cooperative multi-agent tasks, including the challenging StarCraft II micromanagement benchmarks.

References in corpus (19)

• Continuous control with deep reinforcement learning
• Counterfactual Multi-Agent Policy Gradients
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Count-Based Exploration with Neural Density Models
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Self-Imitation Learning
• Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
• MAVEN: Multi-Agent Variational Exploration
• Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics
• Softmax Deep Double Deterministic Policy Gradients
• Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
• Maxmin Q-learning: Controlling the Estimation Bias of Q-learning
• Revisiting the Softmax Bellman Operator: New Benefits and New Perspective
• Evolving Reinforcement Learning Algorithms
• Estimating the Maximum Expected Value: An Analysis of (Nested) Cross Validation and the Maximum Sample Average
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Stabilizing Q Learning Via Soft Mellowmax Operator