Federated Model Distillation with Noise-Free Differential Privacy

Abstract

Conventional federated learning directly averages model weights, which is only possible for collaboration between models with homogeneous architectures. Sharing prediction instead of weight removes this obstacle and eliminates the risk of white-box inference attacks in conventional federated learning. However, the predictions from local models are sensitive and would leak training data privacy to the public. To address this issue, one naive approach is adding the differentially private random noise to the predictions, which however brings a substantial trade-off between privacy budget and model performance. In this paper, we propose a novel framework called FEDMD-NFDP, which applies a Noise-Free Differential Privacy (NFDP) mechanism into a federated model distillation framework. Our extensive experimental results on various datasets validate that FEDMD-NFDP can deliver not only comparable utility and communication efficiency but also provide a noise-free differential privacy guarantee. We also demonstrate the feasibility of our FEDMD-NFDP by considering both IID and non-IID setting, heterogeneous model architectures, and unlabelled public datasets from a different distribution.

accepted by IJCAI-21

References in corpus (5)

• Distilling the Knowledge in a Neural Network
• FedMD: Heterogenous Federated Learning via Model Distillation
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• LDP-FL: Practical Private Aggregation in Federated Learning with Local Differential Privacy
• Differentially Private Deep Learning with Smooth Sensitivity

Cited by in corpus (8)

• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learning
• LDP-FL: Practical Private Aggregation in Federated Learning with Local Differential Privacy
• Secure Deep Graph Generation with Link Differential Privacy
• Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness
• Source Inference Attacks in Federated Learning
• Fair and Differentially Private Distributed Frequency Estimation