Privacy-preserving Decentralized Aggregation for Federated Learning

Abstract

Federated learning is a promising framework for learning over decentralized data spanning multiple regions. This approach avoids expensive central training data aggregation cost and can improve privacy because distributed sites do not have to reveal privacy-sensitive data. In this paper, we develop a privacy-preserving decentralized aggregation protocol for federated learning. We formulate the distributed aggregation protocol with the Alternating Direction Method of Multiplier (ADMM) and examine its privacy weakness. Unlike prior work that use Differential Privacy or homomorphic encryption for privacy, we develop a protocol that controls communication among participants in each round of aggregation to minimize privacy leakage. We establish its privacy guarantee against an honest-but-curious adversary. We also propose an efficient algorithm to construct such a communication pattern, inspired by combinatorial block design theory. Our secure aggregation protocol based on this novel group communication pattern design leads to an efficient algorithm for federated training with privacy guarantees. We evaluate our federated training algorithm on image classification and next-word prediction applications over benchmark datasets with 9 and 15 distributed sites. Evaluation results show that our algorithm performs comparably to the standard centralized federated learning method while preserving privacy; the degradation in test accuracy is only up to 0.73%.

10 pages, 6 figures

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• Federated Learning for Mobile Keyboard Prediction
• On the Linear Convergence of the ADMM in Decentralized Consensus Optimization
• EMNIST: an extension of MNIST to handwritten letters
• A generic framework for privacy preserving deep learning
• HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning
• LEAF: A Benchmark for Federated Settings
• Federated Learning Of Out-Of-Vocabulary Words
• Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms
• Secure Computation for Machine Learning With SPDZ
• QUOTIENT: Two-Party Secure Neural Network Training and Prediction

Cited by in corpus (1)

• Enabling SQL-based Training Data Debugging for Federated Learning