Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use case

Abstract

Deep Reinforcement Learning (DRL) has shown a dramatic improvement in decision-making and automated control problems. Consequently, DRL represents a promising technique to efficiently solve many relevant optimization problems (e.g., routing) in self-driving networks. However, existing DRL-based solutions applied to networking fail to generalize, which means that they are not able to operate properly when applied to network topologies not observed during training. This lack of generalization capability significantly hinders the deployment of DRL technologies in production networks. This is because state-of-the-art DRL-based networking solutions use standard neural networks (e.g., fully connected, convolutional), which are not suited to learn from information structured as graphs. In this paper, we integrate Graph Neural Networks (GNN) into DRL agents and we design a problem specific action space to enable generalization. GNNs are Deep Learning models inherently designed to generalize over graphs of different sizes and structures. This allows the proposed GNN-based DRL agent to learn and generalize over arbitrary network topologies. We test our DRL+GNN agent in a routing optimization use case in optical networks and evaluate it on 180 and 232 unseen synthetic and real-world network topologies respectively. The results show that the DRL+GNN agent is able to outperform state-of-the-art solutions in topologies never seen during training.

arXiv comment12 pages

References in corpus (10)

• Playing Atari with Deep Reinforcement Learning
• Neural Message Passing for Quantum Chemistry
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Interaction Networks for Learning about Objects, Relations and Physics
• Machine Learning for Networking: Workflow, Advances and Opportunities
• RouteNet: Leveraging Graph Neural Networks for network modeling and optimization in SDN
• Knowledge-Defined Networking
• Unveiling the potential of Graph Neural Networks for network modeling and optimization in SDN
• CFR-RL: Traffic Engineering with Reinforcement Learning in SDN
• Fast emergency paths schema to overcome transient link failures in ospf routing

Cited by in corpus (19)

• Graph-based Deep Learning for Communication Networks: A Survey
• Learning Combinatorial Optimization on Graphs: A Survey with Applications to Networking
• Graph Neural Networks: Taxonomy, Advances and Trends
• Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review
• A Survey on Influence Maximization: From an ML-Based Combinatorial Optimization
• A Tutorial on Ultra-Reliable and Low-Latency Communications in 6G: Integrating Domain Knowledge into Deep Learning
• The Graph Neural Networking Challenge: A Worldwide Competition for Education in AI/ML for Networks
• Reinforced Neighborhood Selection Guided Multi-Relational Graph Neural Networks
• Graph-enabled Reinforcement Learning for Time Series Forecasting with Adaptive Intelligence
• Causal Question Answering with Reinforcement Learning
• Multi-Robot Planning on Dynamic Topological Graphs using Mixed-Integer Programming
• Financial Crime & Fraud Detection Using Graph Computing: Application Considerations & Outlook
• Graph Computing for Financial Crime and Fraud Detection: Trends, Challenges and Outlook
• Towards a Robust Transport Network With Self-adaptive Network Digital Twin
• Online Influence Maximization: Concept and Algorithm
• The Power of Communication in a Distributed Multi-Agent System
• RELiQ: Scalable Entanglement Routing via Reinforcement Learning in Quantum Networks
• Learning Cache Coherence Traffic for NoC Routing Design
• Relational Deep Reinforcement Learning for Routing in Wireless Networks