Federated Reinforcement Learning: Techniques, Applications, and Open Challenges

Abstract

This paper presents a comprehensive survey of Federated Reinforcement Learning (FRL), an emerging and promising field in Reinforcement Learning (RL). Starting with a tutorial of Federated Learning (FL) and RL, we then focus on the introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL algorithms can be divided into two categories, i.e. Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition, the existing works on FRL are summarized by application fields, including edge computing, communication, control optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to solving the open problems within FRL.

References in corpus (8)

• Federated Machine Learning: Concept and Applications
• Revisiting Distributed Synchronous SGD
• Massively Parallel Methods for Deep Reinforcement Learning
• Threats to Federated Learning: A Survey
• Efficient Parallel Methods for Deep Reinforcement Learning
• Experience-driven Networking: A Deep Reinforcement Learning based Approach
• Federated Transfer Reinforcement Learning for Autonomous Driving
• Multi-user Resource Control with Deep Reinforcement Learning in IoT Edge Computing

Cited by in corpus (11)

• A Review of Safe Reinforcement Learning Methods for Modern Power Systems
• FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergence
• Differentiated Federated Reinforcement Learning Based Traffic Offloading on Space-Air-Ground Integrated Networks
• Review and Analysis of Recent Advances in Intelligent Network Softwarization for the Internet of Things
• Resilient Control of Networked Microgrids using Vertical Federated Reinforcement Learning: Designs and Real-Time Test-Bed Validations
• Decentralized Federated Reinforcement Learning for User-Centric Dynamic TFDD Control
• Enhancing Cyber Resilience of Networked Microgrids using Vertical Federated Reinforcement Learning
• Combining Federated Learning and Control: A Survey
• Blockchain-assisted Demonstration Cloning for Multi-Agent Deep Reinforcement Learning
• AVDDPG: Federated reinforcement learning applied to autonomous platoon control
• Realistic Urban Traffic Generator using Decentralized Federated Learning for the SUMO simulator