Differential Privacy and Byzantine Resilience in SGD: Do They Add Up?
arXiv:2102.08166
Abstract
This paper addresses the problem of combining Byzantine resilience with privacy in machine learning (ML). Specifically, we study if a distributed implementation of the renowned Stochastic Gradient Descent (SGD) learning algorithm is feasible with both differential privacy (DP) and -Byzantine resilience. To the best of our knowledge, this is the first work to tackle this problem from a theoretical point of view. A key finding of our analyses is that the classical approaches to these two (seemingly) orthogonal issues are incompatible. More precisely, we show that a direct composition of these techniques makes the guarantees of the resulting SGD algorithm depend unfavourably upon the number of parameters of the ML model, making the training of large models practically infeasible. We validate our theoretical results through numerical experiments on publicly-available datasets; showing that it is impractical to ensure DP and Byzantine resilience simultaneously.
References in corpus (10)
- Deep Learning with Differential Privacy
- Federated Learning: Strategies for Improving Communication Efficiency
- Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
- The Hidden Vulnerability of Distributed Learning in Byzantium
- Why gradient clipping accelerates training: A theoretical justification for adaptivity
- Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance
- Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds
- Phocas: dimensional Byzantine-resilient stochastic gradient descent
- Secure Byzantine-Robust Machine Learning
- Byzantine-Resilient Secure Federated Learning