Multi-Agent Trust Region Policy Optimization
arXiv:2010.07916
Abstract
We extend trust region policy optimization (TRPO) to multi-agent reinforcement learning (MARL) problems. We show that the policy update of TRPO can be transformed into a distributed consensus optimization problem for multi-agent cases. By making a series of approximations to the consensus optimization model, we propose a decentralized MARL algorithm, which we call multi-agent TRPO (MATRPO). This algorithm can optimize distributed policies based on local observations and private rewards. The agents do not need to know observations, rewards, policies or value/action-value functions of other agents. The agents only share a likelihood ratio with their neighbors during the training process. The algorithm is fully decentralized and privacy-preserving. Our experiments on two cooperative games demonstrate its robust performance on complicated MARL tasks.
References in corpus (11)
- Trust Region Policy Optimization
- Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
- Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
- Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
- Emergence of Grounded Compositional Language in Multi-Agent Populations
- Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning
- -Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
- Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
- Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning
- Reinforcement Learning and Deep Learning based Lateral Control for Autonomous Driving
- F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
Cited by in corpus (5)
- Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
- The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
- A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
- Coordinated Proximal Policy Optimization
- EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization