Distributed Momentum for Byzantine-resilient Learning

Abstract

Momentum is a variant of gradient descent that has been proposed for its benefits on convergence. In a distributed setting, momentum can be implemented either at the server or the worker side. When the aggregation rule used by the server is linear, commutativity with addition makes both deployments equivalent. Robustness and privacy are however among motivations to abandon linear aggregation rules. In this work, we demonstrate the benefits on robustness of using momentum at the worker side. We first prove that computing momentum at the workers reduces the variance-norm ratio of the gradient estimation at the server, strengthening Byzantine resilient aggregation rules. We then provide an extensive experimental demonstration of the robustness effect of worker-side momentum on distributed SGD.

arXiv commentSource code (for academic use only): https://github.com/LPD-EPFL/ByzantineMomentum

References in corpus (15)

• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Can You Really Backdoor Federated Learning?
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Generalized Byzantine-tolerant SGD
• Byzantine Stochastic Gradient Descent
• Byzantine-Robust Federated Machine Learning through Adaptive Model Averaging
• Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• A Little Is Enough: Circumventing Defenses For Distributed Learning
• Phocas: dimensional Byzantine-resilient stochastic gradient descent
• Momentum-Based Variance Reduction in Non-Convex SGD
• Asynchronous Byzantine Machine Learning (the case of SGD)

Cited by in corpus (4)

• Learning from History for Byzantine Robust Optimization
• Byzantine-Robust Variance-Reduced Federated Learning over Distributed Non-i.i.d. Data
• A Survey on Fault-tolerance in Distributed Optimization and Machine Learning
• Tournesol: A quest for a large, secure and trustworthy database of reliable human judgments