Constrained Differentially Private Federated Learning for Low-bandwidth Devices
arXiv:2103.00342
Abstract
Federated learning becomes a prominent approach when different entities want to learn collaboratively a common model without sharing their training data. However, Federated learning has two main drawbacks. First, it is quite bandwidth inefficient as it involves a lot of message exchanges between the aggregating server and the participating entities. This bandwidth and corresponding processing costs could be prohibitive if the participating entities are, for example, mobile devices. Furthermore, although federated learning improves privacy by not sharing data, recent attacks have shown that it still leaks information about the training data. This paper presents a novel privacy-preserving federated learning scheme. The proposed scheme provides theoretical privacy guarantees, as it is based on Differential Privacy. Furthermore, it optimizes the model accuracy by constraining the model learning phase on few selected weights. Finally, as shown experimentally, it reduces the upstream and downstream bandwidth by up to 99.9% compared to standard federated learning, making it practical for mobile systems.
arXiv commentarXiv admin note: text overlap with arXiv:2011.05578
References in corpus (14)
- Deep Learning with Differential Privacy
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
- Learning Differentially Private Recurrent Language Models
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Differentially Private Federated Learning: A Client Level Perspective
- iDLG: Improved Deep Leakage from Gradients
- Inverting Gradients -- How easy is it to break privacy in federated learning?
- Practical Secure Aggregation for Federated Learning on User-Held Data
- Survey of resampling techniques for improving classification performance in unbalanced datasets
- DSD: Dense-Sparse-Dense Training for Deep Neural Networks
- Rényi Differential Privacy of the Sampled Gaussian Mechanism
- signSGD: Compressed Optimisation for Non-Convex Problems
- LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets