Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents

Abstract

We consider the problem of \emph{fully decentralized} multi-agent reinforcement learning (MARL), where the agents are located at the nodes of a time-varying communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network. Within this setting, the collective goal of the agents is to maximize the globally averaged return over the network through exchanging information with their neighbors. To this end, we propose two decentralized actor-critic algorithms with function approximation, which are applicable to large-scale MARL problems where both the number of states and the number of agents are massively large. Under the decentralized structure, the actor step is performed individually by each agent with no need to infer the policies of others. For the critic step, we propose a consensus update via communication over the network. Our algorithms are fully incremental and can be implemented in an online fashion. Convergence analyses of the algorithms are provided when the value functions are approximated within the class of linear functions. Extensive simulation results with both linear and nonlinear function approximations are presented to validate the proposed algorithms. Our work appears to be the first study of fully decentralized MARL algorithms for networked agents with function approximation, with provable convergence guarantees.

References in corpus (15)

• Continuous control with deep reinforcement learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
• Deep Reinforcement Learning: An Overview
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• An Actor-Critic Algorithm for Sequence Prediction
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Distral: Robust Multitask Reinforcement Learning
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning
• Cooperative Convex Optimization in Networked Systems: Augmented Lagrangian Algorithms with Directed Gossip Communication
• Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning
• Actor-Critic Algorithms for Learning Nash Equilibria in N-player General-Sum Games

Cited by in corpus (72)

• Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligence
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
• Discrete-Time Matrix-Weighted Consensus
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games with Local Control and Global State Information
• Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies
• Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup
• Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning
• Graph Convolutional Reinforcement Learning
• Deep Fictitious Play for Finding Markovian Nash Equilibrium in Multi-Agent Games
• Multi-Agent Reinforcement Learning for Dynamic Ocean Monitoring by a Swarm of Buoys
• Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity
• Settling the Variance of Multi-Agent Policy Gradients
• Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward
• Byzantine-Resilient Decentralized TD Learning with Linear Function Approximation
• A Deep Recurrent Q Network towards Self-adapting Distributed Microservices architecture
• Privacy-Preserving Federated Learning for UAV-Enabled Networks: Learning-Based Joint Scheduling and Resource Management
• Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey
• Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Fast Multi-Agent Temporal-Difference Learning via Homotopy Stochastic Primal-Dual Optimization
• Multi-Agent Trust Region Policy Optimization
• Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent
• Multi-Agent Actor-Critic with Hierarchical Graph Attention Network
• Cooperative Multi-Agent Reinforcement Learning with Partial Observations
• Multi-Agent Deep Reinforcement Learning for Request Dispatching in Distributed-Controller Software-Defined Networking
• Multi-Agent Actor-Critic with Generative Cooperative Policy Network
• Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance
• Stochastic Primal-Dual Q-Learning
• Asynchronous Decentralized Successive Convex Approximation
• Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator
• Finite-Sample Analysis of Decentralized Temporal-Difference Learning with Linear Function Approximation
• A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
• Graph Neural Networks for Decentralized Multi-Robot Submodular Action Selection
• Primal-Dual Distributed Temporal Difference Learning
• A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning
• Agent-Centric Representations for Multi-Agent Reinforcement Learning
• Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition
• PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
• A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning
• Finite-Sample Analysis For Decentralized Batch Multi-Agent Reinforcement Learning With Networked Agents
• Obstacle Avoidance and Navigation Utilizing Reinforcement Learning with Reward Shaping
• Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis
• Gradient play in stochastic games: stationary points, convergence, and sample complexity
• Intelligent Coordination among Multiple Traffic Intersections Using Multi-Agent Reinforcement Learning
• Regret Bounds for Decentralized Learning in Cooperative Multi-Agent Dynamical Systems
• Learning and Management for Internet-of-Things: Accounting for Adaptivity and Scalability
• Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning
• Blind interactive learning of modulation schemes: Multi-agent cooperation without co-design
• Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach
• Finite-Time Error Bounds for Distributed Linear Stochastic Approximation
• Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning
• Compositionality of Linearly Solvable Optimal Control in Networked Multi-Agent Systems
• Online Planning for Decentralized Stochastic Control with Partial History Sharing
• A Convergence Result for Regularized Actor-Critic Methods
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems
• MARLIM: Multi-Agent Reinforcement Learning for Inventory Management
• Exploiting Fast Decaying and Locality in Multi-Agent MDP with Tree Dependence Structure
• Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts
• Provably Efficient Multi-Task Reinforcement Learning with Model Transfer
• Distributed Q-Learning with State Tracking for Multi-agent Networked Control
• Distributed Learning of Average Belief Over Networks Using Sequential Observations
• Multi-Agent Decentralized Belief Propagation on Graphs
• Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge Industrial IoT
• Distributed Safe Learning using an Invariance-based Safety Framework
• Logical Team Q-learning: An approach towards factored policies in cooperative MARL
• Voting-Based Multi-Agent Reinforcement Learning for Intelligent IoT
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Resilient Consensus-based Multi-agent Reinforcement Learning with Function Approximation
• Efficient Querying for Cooperative Probabilistic Commitments
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• The Confluence of Networks, Games and Learning