M${}^3$RL: Mind-aware Multi-agent Management Reinforcement Learning

Abstract

Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward. In this paper, we aim to address this from a different angle. In particular, we consider scenarios where there are self-interested agents (i.e., worker agents) which have their own minds (preferences, intentions, skills, etc.) and can not be dictated to perform tasks they do not wish to do. For achieving optimal coordination among these agents, we train a super agent (i.e., the manager) to manage them by first inferring their minds based on both current and past observations and then initiating contracts to assign suitable tasks to workers and promise to reward them with corresponding bonuses so that they will agree to work together. The objective of the manager is maximizing the overall productivity as well as minimizing payments made to the workers for ad-hoc worker teaming. To train the manager, we propose Mind-aware Multi-agent Management Reinforcement Learning (M^3RL), which consists of agent modeling and policy learning. We have evaluated our approach in two environments, Resource Collection and Crafting, to simulate multi-agent management problems with various task settings and multiple designs for the worker agents. The experimental results have validated the effectiveness of our approach in modeling worker agents' minds online, and in achieving optimal ad-hoc teaming with good generalization and fast adaptation.

ICLR 2019; 18 pages, 12 figures

References in corpus (8)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Designing Neural Network Architectures using Reinforcement Learning
• Learning to reinforcement learn
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• Learned Optimizers that Scale and Generalize
• Machine Theory of Mind
• Universal Successor Representations for Transfer Reinforcement Learning

Cited by in corpus (7)

• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defense
• ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
• PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning
• Robust Multi-agent Counterfactual Prediction
• Modeling Conceptual Understanding in Image Reference Games
• Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning