Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances

Abstract

Multi-agent reinforcement learning (MARL) has long been a significant and everlasting research topic in both machine learning and control. With the recent development of (single-agent) deep RL, there is a resurgence of interests in developing new MARL algorithms, especially those that are backed by theoretical analysis. In this paper, we review some recent advances a sub-area of this topic: decentralized MARL with networked agents. Specifically, multiple agents perform sequential decision-making in a common environment, without the coordination of any central controller. Instead, the agents are allowed to exchange information with their neighbors over a communication network. Such a setting finds broad applications in the control and operation of robots, unmanned vehicles, mobile sensor networks, and smart grid. This review is built upon several our research endeavors in this direction, together with some progresses made by other researchers along the line. We hope this review to inspire the devotion of more research efforts to this exciting yet challenging area.

This is a invited submission to a Special Issue of the Journal of Frontiers of Information Technology & Electronic Engineering (FITEE). Most of the contents are based on the Sec. 4 in our recent overview arXiv:1911.10635, with focus on the setting of decentralized MARL with networked agents

References in corpus (7)

• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Diffusion Strategies Outperform Consensus Strategies for Distributed Estimation over Adaptive Networks
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• Cooperative Convex Optimization in Networked Systems: Augmented Lagrangian Algorithms with Directed Gossip Communication
• Finite-Sample Analysis of Proximal Gradient TD Algorithms
• Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning
• Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learning

Cited by in corpus (9)

• Fast Multi-Agent Temporal-Difference Learning via Homotopy Stochastic Primal-Dual Optimization
• Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective
• Distributed Algorithms for Linearly-Solvable Optimal Control in Networked Multi-Agent Systems
• Scalable Multi-agent Reinforcement Learning Algorithm for Wireless Networks
• Common Information based Approximate State Representations in Multi-Agent Reinforcement Learning
• Context-Aware Deep Q-Network for Decentralized Cooperative Reconnaissance by a Robotic Swarm
• Some Limit Properties of Markov Chains Induced by Stochastic Recursive Algorithms
• Reinforcement Learning based Distributed Control of Dissipative Networked Systems
• Decentralized Cooperative Reinforcement Learning with Hierarchical Information Structure