RODE: Learning Roles to Decompose Multi-Agent Tasks

Abstract

Role-based learning holds the promise of achieving scalable multi-agent learning by decomposing complex tasks using roles. However, it is largely unclear how to efficiently discover such a set of roles. To solve this problem, we propose to first decompose joint action spaces into restricted role action spaces by clustering actions according to their effects on the environment and other agents. Learning a role selector based on action effects makes role discovery much easier because it forms a bi-level learning hierarchy -- the role selector searches in a smaller role space and at a lower temporal resolution, while role policies learn in significantly reduced primitive action-observation spaces. We further integrate information about action effects into the role policies to boost learning efficiency and policy generalization. By virtue of these advances, our method (1) outperforms the current state-of-the-art MARL algorithms on 10 of the 14 scenarios that comprise the challenging StarCraft II micromanagement benchmark and (2) achieves rapid transfer to new environments with three times the number of agents. Demonstrative videos are available at https://sites.google.com/view/rode-marl .

References in corpus (31)

• Dota 2 with Large Scale Deep Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Variational Intrinsic Control
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Multi-Agent Cooperation and the Emergence of (Natural) Language
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• MAVEN: Multi-Agent Variational Exploration
• Learning Multi-Level Hierarchies with Hindsight
• Inequity aversion improves cooperation in intertemporal social dilemmas
• Cooperative and Distributed Reinforcement Learning of Drones for Field Coverage
• Planning with Goal-Conditioned Policies
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Disentangling the independently controllable factors of variation by interacting with the world
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• EMI: Exploration with Mutual Information
• Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning
• Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning
• Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learning
• Learning Robot Skills with Temporal Variational Inference
• Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• Learning Action-Transferable Policy with Action Embedding
• Generalization to New Actions in Reinforcement Learning
• A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

Cited by in corpus (9)

• Advances of Machine Learning in Materials Science: Ideas and Techniques
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning
• SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates
• Model based Multi-agent Reinforcement Learning with Tensor Decompositions
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism