Multi-Agent Trust Region Policy Optimization

Abstract

We extend trust region policy optimization (TRPO) to multi-agent reinforcement learning (MARL) problems. We show that the policy update of TRPO can be transformed into a distributed consensus optimization problem for multi-agent cases. By making a series of approximations to the consensus optimization model, we propose a decentralized MARL algorithm, which we call multi-agent TRPO (MATRPO). This algorithm can optimize distributed policies based on local observations and private rewards. The agents do not need to know observations, rewards, policies or value/action-value functions of other agents. The agents only share a likelihood ratio with their neighbors during the training process. The algorithm is fully decentralized and privacy-preserving. Our experiments on two cooperative games demonstrate its robust performance on complicated MARL tasks.

References in corpus (4)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning
• Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning

Cited by in corpus (5)

• Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
• The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
• A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
• Coordinated Proximal Policy Optimization
• EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization