Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control

Abstract

Traffic signal control (TSC) is a challenging problem within intelligent transportation systems and has been tackled using multi-agent reinforcement learning (MARL). While centralized approaches are often infeasible for large-scale TSC problems, decentralized approaches provide scalability but introduce new challenges, such as partial observability. Communication plays a critical role in decentralized MARL, as agents must learn to exchange information using messages to better understand the system and achieve effective coordination. Deep MARL has been used to enable inter-agent communication by learning communication protocols in a differentiable manner. However, many deep MARL communication frameworks proposed for TSC allow agents to communicate with all other agents at all times, which can add to the existing noise in the system and degrade overall performance. In this study, we propose a communication-based MARL framework for large-scale TSC. Our framework allows each agent to learn a communication policy that dictates "which" part of the message is sent "to whom". In essence, our framework enables agents to selectively choose the recipients of their messages and exchange variable length messages with them. This results in a decentralized and flexible communication mechanism in which agents can effectively use the communication channel only when necessary. We designed two networks, a synthetic $4\times 4$ grid network and a real-world network based on the Pasubio neighborhood in Bologna. Our framework achieved the lowest network congestion compared to related methods, with agents utilizing $\sim 47-65\%$ of the communication channel. Ablation studies further demonstrated the effectiveness of the communication policies learned within our framework.

arXiv commentIEEE Access (2023)

References in corpus (19)

• Categorical Reparameterization with Gumbel-Softmax
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• CoLight: Learning Network-level Cooperation for Traffic Signal Control
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• The Communicative Multiagent Team Decision Problem: Analyzing Teamwork Theories and Models
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Using a Deep Reinforcement Learning Agent for Traffic Signal Control
• Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning
• A Survey on Traffic Signal Control Methods
• IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control
• Multi-agent Reinforcement Learning for Networked System Control
• An Efficient Deep Reinforcement Learning Model for Urban Traffic Control
• Improving Coordination in Small-Scale Multi-Agent Deep Reinforcement Learning through Memory-driven Communication
• Networked Multi-Agent Reinforcement Learning with Emergent Communication
• MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control
• Learning Scalable Multi-Agent Coordination by Spatial Differentiation for Traffic Signal Control