Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
arXiv:1908.02269
Abstract
In multi-agent reinforcement learning, discovering successful collective behaviors is challenging as it requires exploring a joint action space that grows exponentially with the number of agents. While the tractability of independent agent-wise exploration is appealing, this approach fails on tasks that require elaborate group strategies. We argue that coordinating the agents' policies can guide their exploration and we investigate techniques to promote such an inductive bias. We propose two policy regularization methods: TeamReg, which is based on inter-agent action predictability and CoachReg that relies on synchronized behavior selection. We evaluate each approach on four challenging continuous control tasks with sparse rewards that require varying levels of coordination as well as on the discrete action Google Research Football environment. Our experiments show improved performance across many cooperative multi-agent problems. Finally, we analyze the effects of our proposed methods on the policies that our agents learn and show that our methods successfully enforce the qualities that we propose as proxies for coordinated behaviors.
arXiv comment23 pages, 16 figures. This revised version contains additional results and minor edits
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Categorical Reparameterization with Gumbel-Softmax
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- A Survey and Critique of Multiagent Deep Reinforcement Learning
- Counterfactual Multi-Agent Policy Gradients
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Learning Attentional Communication for Multi-Agent Cooperation
- Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
- Emergence of Grounded Compositional Language in Multi-Agent Populations
- Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning
- A Deep Policy Inference Q-Network for Multi-Agent Systems
- Learning to Share and Hide Intentions using Information Regularization
- Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning
- Agent Modeling as Auxiliary Task for Deep Reinforcement Learning