Multiagent Soft Q-Learning

Abstract

Policy gradient methods are often applied to reinforcement learning in continuous multiagent games. These methods perform local search in the joint-action space, and as we show, they are susceptable to a game-theoretic pathology known as relative overgeneralization. To resolve this issue, we propose Multiagent Soft Q-learning, which can be seen as the analogue of applying Q-learning to continuous controls. We compare our method to MADDPG, a state-of-the-art approach, and show that our method achieves better coordination in multiagent cooperative tasks, converging to better local optima in the joint action space.

arXiv commentAccepted in AAAI 18 Spring Symposium

References in corpus (8)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• Learning to Cooperate via Policy Search

Cited by in corpus (19)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Deep Coordination Graphs
• Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
• Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• A Generalized Training Approach for Multiagent Learning
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• Deep Reinforcement Learning for the Heat Transfer Control of Pulsating Impinging Jets
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• A Regularized Opponent Model with Maximum Entropy Objective
• Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning
• Value-Decomposition Multi-Agent Actor-Critics
• Correcting Experience Replay for Multi-Agent Communication
• Multi-Agent Interactions Modeling with Correlated Policies
• Energy-based Surprise Minimization for Multi-Agent Value Factorization
• Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts