ROMA: Multi-Agent Reinforcement Learning with Emergent Roles

Abstract

The role concept provides a useful tool to design and understand complex multi-agent systems, which allows agents with a similar role to share similar behaviors. However, existing role-based methods use prior domain knowledge and predefine role structures and behaviors. In contrast, multi-agent reinforcement learning (MARL) provides flexibility and adaptability, but less efficiency in complex tasks. In this paper, we synergize these two paradigms and propose a role-oriented MARL framework (ROMA). In this framework, roles are emergent, and agents with similar roles tend to share their learning and to be specialized on certain sub-tasks. To this end, we construct a stochastic role embedding space by introducing two novel regularizers and conditioning individual policies on roles. Experiments show that our method can learn specialized, dynamic, and identifiable roles, which help our method push forward the state of the art on the StarCraft II micromanagement benchmark. Demonstrative videos are available at https://sites.google.com/view/romarl/.

arXiv commentThirty-seventh International Conference on Machine Learning (ICML 2020)

References in corpus (16)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• StarCraft II: A New Challenge for Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Emergent Tool Use From Multi-Agent Autocurricula
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• MAVEN: Multi-Agent Variational Exploration
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Learning Fairness in Multi-Agent Systems

Cited by in corpus (5)

• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning