A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
arXiv:2011.00382
Abstract
A fundamental challenge in multiagent reinforcement learning is to learn beneficial behaviors in a shared environment with other simultaneously learning agents. In particular, each agent perceives the environment as effectively non-stationary due to the changing policies of other agents. Moreover, each agent is itself constantly learning, leading to natural non-stationarity in the distribution of experiences encountered. In this paper, we propose a novel meta-multiagent policy gradient theorem that directly accounts for the non-stationary policy dynamics inherent to multiagent learning settings. This is achieved by modeling our gradient updates to consider both an agent's own non-stationary policy dynamics and the non-stationary policy dynamics of other agents in the environment. We show that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently comprises all key aspects of previous state of the art approaches on this topic. We test our method on a diverse suite of multiagent benchmarks and demonstrate a more efficient ability to adapt to new agents as they learn than baseline methods across the full spectrum of mixed incentive, competitive, and cooperative domains.
arXiv commentAccepted to ICML 2021. Code at https://github.com/dkkim93/meta-mapg and Videos at https://sites.google.com/view/meta-mapg/home
References in corpus (18)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
- A Simple Neural Attentive Meta-Learner
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Learning to reinforcement learn
- Mean Field Multi-Agent Reinforcement Learning
- A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
- Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
- Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning
- FACMAC: Factored Multi-Agent Centralised Policy Gradients
- Gradient Surgery for Multi-Task Learning
- Learning to Continually Learn
- Learning Policy Representations in Multiagent Systems
- Multiagent Soft Q-Learning
- Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
- La-MAML: Look-ahead Meta Learning for Continual Learning
- Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning
- Meta-learnt priors slow down catastrophic forgetting in neural networks