Skill Discovery of Coordination in Multi-agent Reinforcement Learning
arXiv:2006.04021
Abstract
Unsupervised skill discovery drives intelligent agents to explore the unknown environment without task-specific reward signal, and the agents acquire various skills which may be useful when the agents adapt to new tasks. In this paper, we propose "Multi-agent Skill Discovery"(MASD), a method for discovering skills for coordination patterns of multiple agents. The proposed method aims to maximize the mutual information between a latent code Z representing skills and the combination of the states of all agents. Meanwhile it suppresses the empowerment of Z on the state of any single agent by adversarial training. In another word, it sets an information bottleneck to avoid empowerment degeneracy. First we show the emergence of various skills on the level of coordination in a general particle multi-agent environment. Second, we reveal that the "bottleneck" prevents skills from collapsing to a single agent and enhances the diversity of learned skills. Finally, we show the pretrained policies have better performance on supervised RL tasks.
References in corpus (10)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Counterfactual Multi-Agent Policy Gradients
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Variational Intrinsic Control
- Variational Option Discovery Algorithms
- MAVEN: Multi-Agent Variational Exploration
- Latent Space Policies for Hierarchical Reinforcement Learning
- Coordinated Multi-Agent Imitation Learning
- Unsupervised Meta-Learning for Reinforcement Learning