Generalized Byzantine-tolerant SGD

Abstract

We propose three new robust aggregation rules for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience properties of these aggregation rules. Empirical analysis shows that the proposed techniques outperform current approaches for realistic use cases and Byzantine attack scenarios.

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Geometric median and robust estimation in Banach spaces
• Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent

Cited by in corpus (40)

• How To Backdoor Federated Learning
• Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attacks
• Learning to Detect Malicious Clients for Robust Federated Learning
• Federated Learning: A Signal Processing Perspective
• Byzantine Stochastic Gradient Descent
• Robust Federated Learning in a Heterogeneous Environment
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• Shielding Federated Learning: Robust Aggregation with Adaptive Client Selection
• A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective
• A Little Is Enough: Circumventing Defenses For Distributed Learning
• SAFELearning: Enable Backdoor Detectability In Federated Learning With Secure Aggregation
• Byzantine-Resilient Decentralized TD Learning with Linear Function Approximation
• Backdoor Attacks on Federated Meta-Learning
• RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets
• Byzantine-Resilient Non-Convex Stochastic Gradient Descent
• Byzantine Fault Tolerant Distributed Linear Regression
• Distributed Momentum for Byzantine-resilient Learning
• DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation
• Cybersecurity in Distributed and Fully-Decentralized Optimization: Distortions, Noise Injection, and ADMM
• Variance Reduced Median-of-Means Estimator for Byzantine-Robust Distributed Inference
• Optimal query complexity for private sequential learning against eavesdropping
• Byzantine-Resilient Stochastic Gradient Descent for Distributed Learning: A Lipschitz-Inspired Coordinate-wise Median Approach
• ByGARS: Byzantine SGD with Arbitrary Number of Attackers
• Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning
• Resilience in Collaborative Optimization: Redundant and Independent Cost Functions
• Byzantine Fault-Tolerance in Peer-to-Peer Distributed Gradient-Descent
• A Survey on Fault-tolerance in Distributed Optimization and Machine Learning
• Byzantine-resilient Decentralized Stochastic Gradient Descent
• Exploiting Heterogeneity in Robust Federated Best-Arm Identification
• FAT: Federated Adversarial Training
• Fast Machine Learning with Byzantine Workers and Servers
• Byzantine-Robust Decentralized Stochastic Optimization over Static and Time-Varying Networks
• Better scalability under potentially heavy-tailed gradients
• Communication-efficient Byzantine-robust distributed learning with statistical guarantee
• Distributed Byzantine Tolerant Stochastic Gradient Descent in the Era of Big Data
• Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments
• Resilient Consensus-based Multi-agent Reinforcement Learning with Function Approximation
• Combining Differential Privacy and Byzantine Resilience in Distributed SGD
• Better scalability under potentially heavy-tailed feedback
• Efficient Byzantine-Resilient Stochastic Gradient Desce