Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance

Abstract

We present Zeno, a technique to make distributed machine learning, particularly Stochastic Gradient Descent (SGD), tolerant to an arbitrary number of faulty workers. Zeno generalizes previous results that assumed a majority of non-faulty nodes; we need assume only one non-faulty worker. Our key idea is to suspect workers that are potentially defective. Since this is likely to lead to false positives, we use a ranking-based preference mechanism. We prove the convergence of SGD for non-convex problems under these scenarios. Experimental results show that Zeno outperforms existing approaches.

arXiv commentICML 2019

Cited by in corpus (26)

• Multi-Armed Bandit Based Client Scheduling for Federated Learning
• A Field Guide to Federated Optimization
• Byzantine-Robust Federated Machine Learning through Adaptive Model Averaging
• IBM Federated Learning: an Enterprise Framework White Paper V0.1
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• Flower: A Friendly Federated Learning Research Framework
• Fedstellar: A Platform for Decentralized Federated Learning
• Secure Byzantine-Robust Machine Learning
• FOCUS: Dealing with Label Quality Disparity in Federated Learning
• FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping
• Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training
• Secure Aggregation with Heterogeneous Quantization in Federated Learning
• Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
• Communication-Efficient Robust Federated Learning Over Heterogeneous Datasets
• Byzantine Resilient Non-Convex SVRG with Distributed Batch Gradient Computations
• Robust Training in High Dimensions via Block Coordinate Geometric Median Descent
• Secure Byzantine-Robust Distributed Learning via Clustering
• Mitigating Sybil Attacks on Differential Privacy based Federated Learning
• FedCom: A Byzantine-Robust Local Model Aggregation Rule Using Data Commitment for Federated Learning
• Byzantine-resilient Decentralized Stochastic Gradient Descent
• Byzantine Fault-Tolerance in Decentralized Optimization under Minimal Redundancy
• MANDERA: Malicious Node Detection in Federated Learning via Ranking
• Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee
• Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients
• Differential Privacy and Byzantine Resilience in SGD: Do They Add Up?
• Zeno++: Robust Fully Asynchronous SGD