Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning

Abstract

We consider the networked multi-agent reinforcement learning (MARL) problem in a fully decentralized setting, where agents learn to coordinate to achieve the joint success. This problem is widely encountered in many areas including traffic control, distributed control, and smart grids. We assume that the reward function for each agent can be different and observed only locally by the agent itself. Furthermore, each agent is located at a node of a communication network and can exchanges information only with its neighbors. Using softmax temporal consistency and a decentralized optimization method, we obtain a principled and data-efficient iterative algorithm. In the first step of each iteration, an agent computes its local policy and value gradients and then updates only policy parameters. In the second step, the agent propagates to its neighbors the messages based on its value function and then updates its own value function. Hence we name the algorithm value propagation. We prove a non-asymptotic convergence rate 1/T with the nonlinear function approximation. To the best of our knowledge, it is the first MARL algorithm with convergence guarantee in the control, off-policy and non-linear function approximation setting. We empirically demonstrate the effectiveness of our approach in experiments.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• Graph Convolutional Reinforcement Learning
• Decomposing Linearly Constrained Nonconvex Problems by a Proximal Primal Dual Approach: Algorithms, Convergence, and Applications

Cited by in corpus (9)

• Multi-agent Reinforcement Learning for Networked System Control
• Graph Convolutional Reinforcement Learning
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Primal-Dual Distributed Temporal Difference Learning
• Foresight of Graph Reinforcement Learning Latent Permutations Learnt by Gumbel Sinkhorn Network
• Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts
• Decentralized Multi-Agents by Imitation of a Centralized Controller
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Resilient Consensus-based Multi-agent Reinforcement Learning with Function Approximation