A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning

Abstract

In this paper, we propose a maximum mutual information (MMI) framework for multi-agent reinforcement learning (MARL) to enable multiple agents to learn coordinated behaviors by regularizing the accumulated return with the mutual information between actions. By introducing a latent variable to induce nonzero mutual information between actions and applying a variational bound, we derive a tractable lower bound on the considered MMI-regularized objective function. Applying policy iteration to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic (VM3-AC), which follows centralized learning with decentralized execution (CTDE). We evaluated VM3-AC for several games requiring coordination, and numerical results show that VM3-AC outperforms MADDPG and other MARL algorithms in multi-agent tasks requiring coordination.

References in corpus (10)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Counterfactual Multi-Agent Policy Gradients
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• MAVEN: Multi-Agent Variational Exploration