Evaluating Differentially Private Machine Learning in Practice

Abstract

Differential privacy is a strong notion for privacy that can be used to prove formal guarantees, in terms of a privacy budget, $\epsilon$, about how much information is leaked by a mechanism. However, implementations of privacy-preserving machine learning often select large values of $\epsilon$ in order to get acceptable utility of the model, with little understanding of the impact of such choices on meaningful privacy. Moreover, in scenarios where iterative learning procedures are used, differential privacy variants that offer tighter analyses are used which appear to reduce the needed privacy budget but present poorly understood trade-offs between privacy and utility. In this paper, we quantify the impact of these choices on privacy in experiments with logistic regression and neural network models. Our main finding is that there is a huge gap between the upper bounds on privacy loss that can be guaranteed, even with advanced mechanisms, and the effective privacy loss that can be measured using current inference attacks. Current mechanisms for differentially private machine learning rarely offer acceptable utility-privacy trade-offs with guarantees for complex learning tasks: settings that provide limited accuracy loss provide meaningless privacy guarantees, and settings that provide strong privacy guarantees result in useless models. Code for the experiments can be found here: https://github.com/bargavj/EvaluatingDPML

arXiv commentRevised version of a paper in USENIX Security 2019

References in corpus (17)

• Deep Learning with Differential Privacy
• ADADELTA: An Adaptive Learning Rate Method
• Renyi Differential Privacy
• How To Backdoor Federated Learning
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Differentially Private Federated Learning: A Client Level Perspective
• Protection Against Reconstruction and Its Applications in Private Federated Learning
• Support Vector Machines under Adversarial Label Contamination
• Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Generative Poisoning Attack Method Against Neural Networks
• Efficient Deep Learning on Multi-Source Private Data
• Towards Measuring Membership Privacy
• Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures
• Private Empirical Risk Minimization Beyond the Worst Case: The Effect of the Constraint Set Geometry
• Privacy-Preserving Distributed Deep Learning for Clinical Data
• Privacy-Preserving Collaborative Deep Learning with Unreliable Participants

Cited by in corpus (21)

• Inverting Gradients -- How easy is it to break privacy in federated learning?
• When Machine Unlearning Jeopardizes Privacy
• How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
• A Comprehensive Survey on Local Differential Privacy Toward Data Statistics and Analysis
• A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainability
• Privacy Accounting and Quality Control in the Sage Differentially Private ML Platform
• Not one but many Tradeoffs: Privacy Vs. Utility in Differentially Private Machine Learning
• Sampling Attacks: Amplification of Membership Inference Attacks by Repeated Queries
• Fast-adapting and Privacy-preserving Federated Recommender System
• Fault Tolerance of Neural Networks in Adversarial Settings
• Weighted Distributed Differential Privacy ERM: Convex and Non-convex
• On the Effectiveness of Regularization Against Membership Inference Attacks
• On the Privacy Risks of Cell-Based NAS Architectures
• Source Inference Attacks in Federated Learning
• Correlation inference attacks against machine learning models
• The Trade-Offs of Private Prediction
• Utility/Privacy Trade-off through the lens of Optimal Transport
• DPack: Efficiency-Oriented Privacy Budget Scheduling
• Investigating Trade-offs in Utility, Fairness and Differential Privacy in Neural Networks
• Secure Metric Learning via Differential Pairwise Privacy
• Reducing Risk of Model Inversion Using Privacy-Guided Training