Do I Get the Privacy I Need? Benchmarking Utility in Differential Privacy Libraries

Abstract

An increasing number of open-source libraries promise to bring differential privacy to practice, even for non-experts. This paper studies five libraries that offer differentially private analytics: Google DP, SmartNoise, diffprivlib, diffpriv, and Chorus. We compare these libraries qualitatively (capabilities, features, and maturity) and quantitatively (utility and scalability) across four analytics queries (count, sum, mean, and variance) executed on synthetic and real-world datasets. We conclude that these libraries provide similar utility (except in some notable scenarios). However, there are significant differences in the features provided, and we find that no single library excels in all areas. Based on our results, we provide guidance for practitioners to help in choosing a suitable library, guidance for library designers to enhance their software, and guidance for researchers on open challenges in differential privacy tools for non-experts.

arXiv comment13 pages, 12 figures, 15 tables, and 1 algorithm

References in corpus (9)

• Deep Learning with Differential Privacy
• Statistical applications of the multivariate skew-normal distribution
• Privacy in the Internet of Things: Threats and Challenges
• Local Differential Privacy and Its Applications: A Comprehensive Survey
• Towards matching user mobility traces in large-scale datasets
• Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds
• The Discrete Gaussian for Differential Privacy
• Pain-Free Random Differential Privacy with Sensitivity Sampling
• Privacy and Utility Tradeoff in Approximate Differential Privacy