Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee
arXiv:2110.14074
Abstract
The growing literature of Federated Learning (FL) has recently inspired Federated Reinforcement Learning (FRL) to encourage multiple agents to federatively build a better decision-making policy without sharing raw trajectories. Despite its promising applications, existing works on FRL fail to I) provide theoretical analysis on its convergence, and II) account for random system failures and adversarial attacks. Towards this end, we propose the first FRL framework the convergence of which is guaranteed and tolerant to less than half of the participating agents being random system failures or adversarial attackers. We prove that the sample efficiency of the proposed framework is guaranteed to improve with the number of agents and is able to account for such potential failures or attacks. All theoretical results are empirically verified on various RL benchmark tasks.
Published at NeurIPS 2021. Extended version with proofs and additional experimental details and results. New version changes: reduced file size of figures; added a diagram illustrating the problem setting; added link to code on GitHub; modified proof for Theorem 6 (highlighted in red)
References in corpus (6)
- Federated Optimization: Distributed Machine Learning for On-Device Intelligence
- Challenges of Real-World Reinforcement Learning
- Decentralized Data Fusion and Active Sensing with Mobile Sensors for Modeling and Predicting Spatiotemporal Traffic Phenomena
- Federated Multi-armed Bandits with Personalization
- Byzantine Resilient Non-Convex SVRG with Distributed Batch Gradient Computations
- Differentially Private Federated Bayesian Optimization with Distributed Exploration