Divergence-Regularized Multi-Agent Actor-Critic

Abstract

Entropy regularization is a popular method in reinforcement learning (RL). Although it has many advantages, it alters the RL objective of the original Markov Decision Process (MDP). Though divergence regularization has been proposed to settle this problem, it cannot be trivially applied to cooperative multi-agent reinforcement learning (MARL). In this paper, we investigate divergence regularization in cooperative MARL and propose a novel off-policy cooperative MARL framework, divergence-regularized multi-agent actor-critic (DMAC). Theoretically, we derive the update rule of DMAC which is naturally off-policy and guarantees monotonic policy improvement and convergence in both the original MDP and divergence-regularized MDP. We also give a bound of the discrepancy between the converged policy and optimal policy in the original MDP. DMAC is a flexible framework and can be combined with many existing MARL algorithms. Empirically, we evaluate DMAC in a didactic stochastic game and StarCraft Multi-Agent Challenge and show that DMAC substantially improves the performance of existing MARL algorithms.

arXiv commentICML 2022, 24 pages, 10 figures

References in corpus (14)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Counterfactual Multi-Agent Policy Gradients
• Reinforcement Learning with Deep Energy-Based Policies
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
• A unified view of entropy-regularized Markov decision processes
• If MaxEnt RL is the Answer, What is the Question?
• A Regularized Approach to Sparse Optimal Policy in Reinforcement Learning
• Divergence-Augmented Policy Optimization
• Stable Policy Optimization via Off-Policy Divergence Regularization