Voting-Based Multi-Agent Reinforcement Learning for Intelligent IoT

Abstract

The recent success of single-agent reinforcement learning (RL) in Internet of things (IoT) systems motivates the study of multi-agent reinforcement learning (MARL), which is more challenging but more useful in large-scale IoT. In this paper, we consider a voting-based MARL problem, in which the agents vote to make group decisions and the goal is to maximize the globally averaged returns. To this end, we formulate the MARL problem based on the linear programming form of the policy optimization problem and propose a distributed primal-dual algorithm to obtain the optimal solution. We also propose a voting mechanism through which the distributed learning achieves the same sublinear convergence rate as centralized learning. In other words, the distributed decision making does not slow down the process of achieving global consensus on optimality. Lastly, we verify the convergence of our proposed algorithm with numerical simulations and conduct case studies in practical multi-agent IoT systems.

arXiv commentPublished at IEEE Internet of Things Journal

References in corpus (16)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Placement Optimization of UAV-Mounted Mobile Base Stations
• Applications of Deep Learning and Reinforcement Learning to Biological Data
• Cellular-Connected UAVs over 5G: Deep Reinforcement Learning for Interference Management
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Mean Field Multi-Agent Reinforcement Learning
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
• Stochastic Variance Reduction Methods for Policy Evaluation
• Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
• Primal-Dual $\pi$ Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problems
• Scalable Bilinear $\pi$ Learning Using State and Action Features