Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey

Abstract

Future Internet involves several emerging technologies such as 5G and beyond 5G networks, vehicular networks, unmanned aerial vehicle (UAV) networks, and Internet of Things (IoTs). Moreover, future Internet becomes heterogeneous and decentralized with a large number of involved network entities. Each entity may need to make its local decision to improve the network performance under dynamic and uncertain network environments. Standard learning algorithms such as single-agent Reinforcement Learning (RL) or Deep Reinforcement Learning (DRL) have been recently used to enable each network entity as an agent to learn an optimal decision-making policy adaptively through interacting with the unknown environments. However, such an algorithm fails to model the cooperations or competitions among network entities, and simply treats other entities as a part of the environment that may result in the non-stationarity issue. Multi-agent Reinforcement Learning (MARL) allows each network entity to learn its optimal policy by observing not only the environments, but also other entities' policies. As a result, MARL can significantly improve the learning efficiency of the network entities, and it has been recently used to solve various issues in the emerging networks. In this paper, we thus review the applications of MARL in the emerging networks. In particular, we provide a tutorial of MARL and a comprehensive survey of applications of MARL in next generation Internet. In particular, we first introduce single-agent RL and MARL. Then, we review a number of applications of MARL to solve emerging issues in future Internet. The issues consist of network access, transmit power control, computation offloading, content caching, packet routing, trajectory design for UAV-aided networks, and network security issues.

References in corpus (23)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Prioritized Experience Replay
• Fractional Programming for Communication Systems--Part I: Power Control and Beamforming
• Dueling Network Architectures for Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Deep Reinforcement Learning: An Overview
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Counterfactual Multi-Agent Policy Gradients
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Mean Field Multi-Agent Reinforcement Learning
• The Complexity of Decentralized Control of Markov Decision Processes
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• A Deep-Reinforcement Learning Approach for Software-Defined Networking Routing Optimization
• Multi-Drone 3D Trajectory Planning and Scheduling in Drone Assisted Radio Access Networks
• Reward Design in Cooperative Multi-agent Reinforcement Learning for Packet Routing
• Robust and Scalable Routing with Multi-Agent Deep Reinforcement Learning for MANETs
• Cache-enabled Wireless Networks with Opportunistic Interference Alignment
• Dynamic Coded Caching in Wireless Networks Using Multi-Agent Reinforcement Learning