Byzantine-resilient Decentralized Stochastic Gradient Descent

Abstract

Decentralized learning has gained great popularity to improve learning efficiency and preserve data privacy. Each computing node makes equal contribution to collaboratively learn a Deep Learning model. The elimination of centralized Parameter Servers (PS) can effectively address many issues such as privacy, performance bottleneck and single-point-failure. However, how to achieve Byzantine Fault Tolerance in decentralized learning systems is rarely explored, although this problem has been extensively studied in centralized systems. In this paper, we present an in-depth study towards the Byzantine resilience of decentralized learning systems with two contributions. First, from the adversarial perspective, we theoretically illustrate that Byzantine attacks are more dangerous and feasible in decentralized learning systems: even one malicious participant can arbitrarily alter the models of other participants by sending carefully crafted updates to its neighbors. Second, from the defense perspective, we propose UBAR, a novel algorithm to enhance decentralized learning with Byzantine Fault Tolerance. Specifically, UBAR provides a Uniform Byzantine-resilient Aggregation Rule for benign nodes to select the useful parameter updates and filter out the malicious ones in each training iteration. It guarantees that each benign node in a decentralized system can train a correct model under very strong Byzantine attacks with an arbitrary number of faulty nodes. We conduct extensive experiments on standard image classification tasks and the results indicate that UBAR can effectively defeat both simple and sophisticated Byzantine attacks with higher performance efficiency than existing solutions.

References in corpus (15)

• Towards Federated Learning at Scale: System Design
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Distributed Deep Learning Model for Intelligent Video Surveillance Systems with Edge Computing
• Communication Compression for Decentralized Training
• Generalized Byzantine-tolerant SGD
• Byzantine-Robust Federated Machine Learning through Adaptive Model Averaging
• Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance
• A Little Is Enough: Circumventing Defenses For Distributed Learning
• COLA: Decentralized Linear Learning
• Decentralized Bayesian Learning over Graphs
• Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks
• Robust Learning from Untrusted Sources
• Personalized Federated Deep Learning for Pain Estimation From Face Images
• Distributed Dynamic Map Fusion via Federated Learning for Intelligent Networked Vehicles

Cited by in corpus (3)

• Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training
• Privacy-preserving Collaborative Learning with Automatic Transformation Search
• Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments