ByGARS: Byzantine SGD with Arbitrary Number of Attackers

Abstract

We propose two novel stochastic gradient descent algorithms, ByGARS and ByGARS++, for distributed machine learning in the presence of any number of Byzantine adversaries. In these algorithms, reputation scores of workers are computed using an auxiliary dataset at the server. This reputation score is then used for aggregating the gradients for stochastic gradient descent. The computational complexity of ByGARS++ is the same as the usual distributed stochastic gradient descent method with only an additional inner product computation in every iteration. We show that using these reputation scores for gradient aggregation is robust to any number of multiplicative noise Byzantine adversaries and use two-timescale stochastic approximation theory to prove convergence for strongly convex loss functions. We demonstrate the effectiveness of the algorithms for non-convex learning problems using MNIST and CIFAR-10 datasets against almost all state-of-the-art Byzantine attacks. We also show that the proposed algorithms are robust to multiple different types of attacks at the same time.

arXiv comment16 pages, 8 figures. Added more empirical results against state-of-the-art attacks

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Federated Learning: Strategies for Improving Communication Efficiency
• Towards Federated Learning at Scale: System Design
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Train faster, generalize better: Stability of stochastic gradient descent
• Generalized Byzantine-tolerant SGD
• Byzantine Stochastic Gradient Descent
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation
• Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating
• A Little Is Enough: Circumventing Defenses For Distributed Learning
• Byzantine-Resilient Stochastic Gradient Descent for Distributed Learning: A Lipschitz-Inspired Coordinate-wise Median Approach

Cited by in corpus (4)

• A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learning
• Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training
• Secure Aggregation with Heterogeneous Quantization in Federated Learning
• Secure Distributed Training at Scale