Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning

Abstract

This paper studies a distributed policy gradient in collaborative multi-agent reinforcement learning (MARL), where agents over a communication network aim to find the optimal policy to maximize the average of all agents' local returns. Due to the non-concave performance function of policy gradient, the existing distributed stochastic optimization methods for convex problems cannot be directly used for policy gradient in MARL. This paper proposes a distributed policy gradient with variance reduction and gradient tracking to address the high variances of policy gradient, and utilizes importance weight to solve the {distribution shift} problem in the sampling process. We then provide an upper bound on the mean-squared stationary gap, which depends on the number of iterations, the mini-batch size, the epoch size, the problem parameters, and the network topology. We further establish the sample and communication complexity to obtain an $\epsilon$-approximate stationary point. Numerical experiments are performed to validate the effectiveness of the proposed algorithm.

References in corpus (11)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• DSA: Decentralized Double Stochastic Averaging Gradient Algorithm
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
• Variance-Reduced Decentralized Stochastic Optimization with Accelerated Convergence
• Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning
• A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
• Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk Minimization
• Stabilized SVRG: Simple Variance Reduction for Nonconvex Optimization