Off-Policy Multi-Agent Decomposed Policy Gradients

Abstract

Multi-agent policy gradient (MAPG) methods recently witness vigorous progress. However, there is a significant performance discrepancy between MAPG methods and state-of-the-art multi-agent value-based approaches. In this paper, we investigate causes that hinder the performance of MAPG algorithms and present a multi-agent decomposed policy gradient method (DOP). This method introduces the idea of value function decomposition into the multi-agent actor-critic framework. Based on this idea, DOP supports efficient off-policy learning and addresses the issue of centralized-decentralized mismatch and credit assignment in both discrete and continuous action spaces. We formally show that DOP critics have sufficient representational capability to guarantee convergence. In addition, empirical evaluations on the StarCraft II micromanagement benchmark and multi-agent particle environments demonstrate that DOP significantly outperforms both state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms. Demonstrative videos are available at https://sites.google.com/view/dop-mapg/.

References in corpus (22)

• Categorical Reparameterization with Gumbel-Softmax
• Addressing Function Approximation Error in Actor-Critic Methods
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Dota 2 with Large Scale Deep Reinforcement Learning
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Counterfactual Multi-Agent Policy Gradients
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Multi-Agent Cooperation and the Emergence of (Natural) Language
• Inequity aversion improves cooperation in intertemporal social dilemmas
• Off-Policy Actor-Critic
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning
• A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

Cited by in corpus (12)

• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• A Bayesian Framework for Digital Twin-Based Control, Monitoring, and Data Collection in Wireless Systems
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Settling the Variance of Multi-Agent Policy Gradients
• Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning
• Difference Rewards Policy Gradients
• Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment
• Scalable Multi-agent Reinforcement Learning Algorithm for Wireless Networks
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients