Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee

Abstract

The growing literature of Federated Learning (FL) has recently inspired Federated Reinforcement Learning (FRL) to encourage multiple agents to federatively build a better decision-making policy without sharing raw trajectories. Despite its promising applications, existing works on FRL fail to I) provide theoretical analysis on its convergence, and II) account for random system failures and adversarial attacks. Towards this end, we propose the first FRL framework the convergence of which is guaranteed and tolerant to less than half of the participating agents being random system failures or adversarial attackers. We prove that the sample efficiency of the proposed framework is guaranteed to improve with the number of agents and is able to account for such potential failures or attacks. All theoretical results are empirically verified on various RL benchmark tasks.

arXiv commentPublished at NeurIPS 2021. Extended version with proofs and additional experimental details and results. New version changes: reduced file size of figures; added a diagram illustrating the problem setting; added link to code on GitHub; modified proof for Theorem 6 (highlighted in red)

References in corpus (21)

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems
• Challenges of Real-World Reinforcement Learning
• Byzantine Stochastic Gradient Descent
• Non-convex Finite-Sum Optimization Via SCSG Methods
• Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance
• Decentralized Data Fusion and Active Sensing with Mobile Sensors for Modeling and Predicting Spatiotemporal Traffic Phenomena
• Differentially-Private Federated Linear Bandits
• Collaborative Machine Learning with Incentive-Aware Model Rewards
• Policy Optimization via Importance Sampling
• Less than a Single Pass: Stochastically Controlled Stochastic Gradient Method
• Federated Multi-armed Bandits with Personalization
• Stochastic Variance-Reduced Policy Gradient
• Byzantine Resilient Non-Convex SVRG with Distributed Batch Gradient Computations
• Stochastic Variance Reduction for Policy Gradient Estimation
• Differentially Private Federated Bayesian Optimization with Distributed Exploration