Phocas: dimensional Byzantine-resilient stochastic gradient descent
arXiv:1805.09682
Abstract
We propose a novel robust aggregation rule for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience of the proposed aggregation rules. Empirical analysis shows that the proposed techniques outperform current approaches for realistic use cases and Byzantine attack scenarios.
arXiv commentSubmitted to NIPS 2018. arXiv admin note: substantial text overlap with arXiv:1802.10116
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
- Byzantine Stochastic Gradient Descent
- Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
- Variants of RMSProp and Adagrad with Logarithmic Regret Bounds
Cited by in corpus (13)
- Secure Byzantine-Robust Machine Learning
- Byzantine-Resilient Decentralized TD Learning with Linear Function Approximation
- RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets
- Robust Federated Recommendation System
- Distributed Momentum for Byzantine-resilient Learning
- Byzantine Resilient Non-Convex SVRG with Distributed Batch Gradient Computations
- Byzantine-Resilient Stochastic Gradient Descent for Distributed Learning: A Lipschitz-Inspired Coordinate-wise Median Approach
- A Survey on Fault-tolerance in Distributed Optimization and Machine Learning
- SLSGD: Secure and Efficient Distributed On-device Machine Learning
- On Robust Mean Estimation under Coordinate-level Corruption
- Fast Machine Learning with Byzantine Workers and Servers
- Combining Differential Privacy and Byzantine Resilience in Distributed SGD
- Differential Privacy and Byzantine Resilience in SGD: Do They Add Up?