Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPG

Abstract

Modelling and exploiting teammates' policies in cooperative multi-agent systems have long been an interest and also a big challenge for the reinforcement learning (RL) community. The interest lies in the fact that if the agent knows the teammates' policies, it can adjust its own policy accordingly to arrive at proper cooperations; while the challenge is that the agents' policies are changing continuously due to they are learning concurrently, which imposes difficulty to model the dynamic policies of teammates accurately. In this paper, we present \emph{ATTention Multi-Agent Deep Deterministic Policy Gradient} (ATT-MADDPG) to address this challenge. ATT-MADDPG extends DDPG, a single-agent actor-critic RL method, with two special designs. First, in order to model the teammates' policies, the agent should get access to the observations and actions of teammates. ATT-MADDPG adopts a centralized critic to collect such information. Second, to model the teammates' policies using the collected information in an effective way, ATT-MADDPG enhances the centralized critic with an attention mechanism. This attention mechanism introduces a special structure to explicitly model the dynamic joint policy of teammates, making sure that the collected information can be processed efficiently. We evaluate ATT-MADDPG on both benchmark tasks and the real-world packet routing tasks. Experimental results show that it not only outperforms the state-of-the-art RL-based methods and rule-based methods by a large margin, but also achieves better performance in terms of scalability and robustness.

arXiv commentAttention-based Multi-agent DDPG. Experimental results show that it not only outperforms the state-of-the-art RL-based methods and rule-based methods by a large margin, but also achieves better performance in terms of scalability and robustness

References in corpus (7)

• Continuous control with deep reinforcement learning
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Dueling Network Architectures for Deep Reinforcement Learning
• Opponent Modeling in Deep Reinforcement Learning
• A Multi-Agent, Policy-Gradient approach to Network Routing
• Parameter Sharing Deep Deterministic Policy Gradient for Cooperative Multi-agent Reinforcement Learning
• ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning

Cited by in corpus (9)

• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Reward Design in Cooperative Multi-agent Reinforcement Learning for Packet Routing
• Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling
• Learning Agent Communication under Limited Bandwidth by Message Pruning
• Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning
• Attentional Policies for Cross-Context Multi-Agent Reinforcement Learning
• SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competition
• Learning Altruistic Behaviours in Reinforcement Learning without External Rewards
• SA-MATD3:Self-attention-based multi-agent continuous control method in cooperative environments