DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM

Abstract

Machine learning (ML) models trained by differentially private stochastic gradient descent (DP-SGD) have much lower utility than the non-private ones. To mitigate this degradation, we propose a DP Laplacian smoothing SGD (DP-LSSGD) to train ML models with differential privacy (DP) guarantees. At the core of DP-LSSGD is the Laplacian smoothing, which smooths out the Gaussian noise used in the Gaussian mechanism. Under the same amount of noise used in the Gaussian mechanism, DP-LSSGD attains the same DP guarantee, but in practice, DP-LSSGD makes training both convex and nonconvex ML models more stable and enables the trained models to generalize better. The proposed algorithm is simple to implement and the extra computational complexity and memory overhead compared with DP-SGD are negligible. DP-LSSGD is applicable to train a large variety of ML models, including DNNs. The code is available at \url{https://github.com/BaoWangMath/DP-LSSGD}.

arXiv comment21 pages, 7 figures

References in corpus (7)

• Deep Learning with Differential Privacy
• Renyi Differential Privacy
• Scalable Private Learning with PATE
• Learning with Differential Privacy: Stability, Learnability and the Sufficiency and Necessity of ERM Principle
• Laplacian Smoothing Gradient Descent
• Private Topic Modeling
• Local Differential Privacy for Physical Sensor Data and Sparse Recovery

Cited by in corpus (5)

• Differentially Private Federated Learning with Laplacian Smoothing
• Private and Communication-Efficient Edge Learning: A Sparse Differential Gaussian-Masking Distributed SGD Approach
• DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
• Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo
• Selective Differential Privacy for Language Modeling