Federated Reinforcement Learning: Techniques, Applications, and Open Challenges

Abstract

This paper presents a comprehensive survey of Federated Reinforcement Learning (FRL), an emerging and promising field in Reinforcement Learning (RL). Starting with a tutorial of Federated Learning (FL) and RL, we then focus on the introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL algorithms can be divided into two categories, i.e. Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition, the existing works on FRL are summarized by application fields, including edge computing, communication, control optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to solving the open problems within FRL.

References in corpus (32)

• Continuous control with deep reinforcement learning
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Asynchronous Methods for Deep Reinforcement Learning
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Federated Machine Learning: Concept and Applications
• Revisiting Distributed Synchronous SGD
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Prioritized Experience Replay
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Semi-supervised Deep Reinforcement Learning in Support of IoT and Smart City Services
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Massively Parallel Methods for Deep Reinforcement Learning
• Mitigating Sybils in Federated Learning Poisoning
• Distributed Distributional Deterministic Policy Gradients
• Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems
• Threats to Federated Learning: A Survey
• Memory-based control with recurrent neural networks
• Adaptive Federated Learning in Resource Constrained Edge Computing Systems
• Efficient Parallel Methods for Deep Reinforcement Learning
• On Improving Deep Reinforcement Learning for POMDPs
• Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPG
• Experience-driven Networking: A Deep Reinforcement Learning based Approach
• Federated Transfer Reinforcement Learning for Autonomous Driving
• QMDP-Net: Deep Learning for Planning under Partial Observability
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Multi-user Resource Control with Deep Reinforcement Learning in IoT Edge Computing

Cited by in corpus (11)

• A Review of Safe Reinforcement Learning Methods for Modern Power Systems
• FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergence
• Differentiated Federated Reinforcement Learning Based Traffic Offloading on Space-Air-Ground Integrated Networks
• Review and Analysis of Recent Advances in Intelligent Network Softwarization for the Internet of Things
• Resilient Control of Networked Microgrids using Vertical Federated Reinforcement Learning: Designs and Real-Time Test-Bed Validations
• Decentralized Federated Reinforcement Learning for User-Centric Dynamic TFDD Control
• Enhancing Cyber Resilience of Networked Microgrids using Vertical Federated Reinforcement Learning
• Combining Federated Learning and Control: A Survey
• Blockchain-assisted Demonstration Cloning for Multi-Agent Deep Reinforcement Learning
• AVDDPG: Federated reinforcement learning applied to autonomous platoon control
• Realistic Urban Traffic Generator using Decentralized Federated Learning for the SUMO simulator