Robust Aggregation for Federated Learning

Abstract

Federated learning is the centralized training of statistical models from decentralized data on mobile devices while preserving the privacy of each device. We present a robust aggregation approach to make federated learning robust to settings when a fraction of the devices may be sending corrupted updates to the server. The approach relies on a robust aggregation oracle based on the geometric median, which returns a robust aggregate using a constant number of iterations of a regular non-robust averaging oracle. The robust aggregation oracle is privacy-preserving, similar to the non-robust secure average oracle it builds upon. We establish its convergence for least squares estimation of additive models. We provide experimental results with linear models and deep networks for three tasks in computer vision and natural language processing. The robust aggregation approach is agnostic to the level of corruption; it outperforms the classical aggregation approach in terms of robustness when the level of corruption is high, while being competitive in the regime of low corruption. Two variants, a faster one with one-step robust aggregation and another one with on-device personalization, round off the paper.

References in corpus (7)

• Federated Learning: Challenges, Methods, and Future Directions
• Towards Federated Learning at Scale: System Design
• EMNIST: an extension of MNIST to handwritten letters
• Agnostic Federated Learning
• Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attacks
• Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System
• Study of Robust Diffusion Recursive Least Squares Algorithms with Side Information for Networked Agents

Cited by in corpus (46)

• From Distributed Machine Learning to Federated Learning: A Survey
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Adaptive Personalized Federated Learning
• Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges
• Ditto: Fair and Robust Federated Learning Through Personalization
• Personalized Federated Learning with Moreau Envelopes
• A Field Guide to Federated Optimization
• FastSecAgg: Scalable Secure Aggregation for Privacy-Preserving Federated Learning
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
• Secure Byzantine-Robust Machine Learning
• Auto-weighted Robust Federated Learning with Corrupted Data Sources
• Studying the Robustness of Anti-adversarial Federated Learning Models Detecting Cyberattacks in IoT Spectrum Sensors
• Learning from History for Byzantine Robust Optimization
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training
• Robust Federated Recommendation System
• Byzantines can also Learn from History: Fall of Centered Clipping in Federated Learning
• Incentives for Federated Learning: a Hypothesis Elicitation Approach
• COVID-19 Imaging Data Privacy by Federated Learning Design: A Theoretical Framework
• Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
• SPECTRE: Defending Against Backdoor Attacks Using Robust Statistics
• From Federated Learning to Federated Neural Architecture Search: A Survey
• BlockFLA: Accountable Federated Learning via Hybrid Blockchain Architecture
• Secure Byzantine-Robust Distributed Learning via Clustering
• Robust Training in High Dimensions via Block Coordinate Geometric Median Descent
• A Survey on Fault-tolerance in Distributed Optimization and Machine Learning
• Mitigating Sybil Attacks on Differential Privacy based Federated Learning
• ADI: Adversarial Dominating Inputs in Vertical Federated Learning Systems
• Federated Learning and Wireless Communications
• Faithful Edge Federated Learning: Scalability and Privacy
• Robust Federated Learning with Attack-Adaptive Aggregation
• Byzantine Resilient Distributed Multi-Task Learning
• Towards Bidirectional Protection in Federated Learning
• Adversarial Robustness through Bias Variance Decomposition: A New Perspective for Federated Learning
• Federated Multi-Armed Bandits Under Byzantine Attacks
• Reliability and Performance Assessment of Federated Learning on Clinical Benchmark Data
• ABC-FL: Anomalous and Benign client Classification in Federated Learning
• Learning-to-learn non-convex piecewise-Lipschitz functions
• Robust Federated Learning by Mixture of Experts
• Probabilistic Inference for Learning from Untrusted Sources
• Federated Learning from Small Datasets
• Secure Distributed Training at Scale
• Two-Bit Aggregation for Communication Efficient and Differentially Private Federated Learning
• Byzantine-Resilient Federated Machine Learning via Over-the-Air Computation