DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
arXiv:2103.01496
Abstract
Deep learning techniques have achieved remarkable performance in wide-ranging tasks. However, when trained on privacy-sensitive datasets, the model parameters may expose private information in training data. Prior attempts for differentially private training, although offering rigorous privacy guarantees, lead to much lower model performance than the non-private ones. Besides, different runs of the same training algorithm produce models with large performance variance. To address these issues, we propose DPlis--Differentially Private Learning wIth Smoothing. The core idea of DPlis is to construct a smooth loss function that favors noise-resilient models lying in large flat regions of the loss landscape. We provide theoretical justification for the utility improvements of DPlis. Extensive experiments also demonstrate that DPlis can effectively boost model quality and training stability under a given privacy budget.
arXiv commentThe 21st Privacy Enhancing Technologies Symposium (PETS), 2021
References in corpus (21)
- Deep Learning with Differential Privacy
- Renyi Differential Privacy
- Learning Differentially Private Recurrent Language Models
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
- Scalable Private Learning with PATE
- Understanding Membership Inferences on Well-Generalized Learning Models
- Subsampled Rényi Differential Privacy and Analytical Moments Accountant
- Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning
- Differentially Private Learning with Adaptive Clipping
- Conversational Contextual Cues: The Case of Personalization and History for Response Ranking
- Private Empirical Risk Minimization Beyond the Worst Case: The Effect of the Constraint Set Geometry
- SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning
- Tempered Sigmoid Activations for Deep Learning with Differential Privacy
- Adversarial Neural Network Inversion via Auxiliary Knowledge Alignment
- DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM
- Efficient Private ERM for Smooth Objectives
- RIGA: Covert and Robust White-Box Watermarking of Deep Neural Networks
- Improving Robustness to Model Inversion Attacks via Mutual Information Regularization
- Differentially Private Deep Learning with Direct Feedback Alignment
- Differentially Private Regression and Classification with Sparse Gaussian Processes