Deep Learning with Label Differential Privacy
arXiv:2102.06062
Abstract
The Randomized Response (RR) algorithm is a classical technique to improve robustness in survey aggregation, and has been widely adopted in applications with differential privacy guarantees. We propose a novel algorithm, Randomized Response with Prior (RRWithPrior), which can provide more accurate results while maintaining the same level of privacy guaranteed by RR. We then apply RRWithPrior to learn neural networks with label differential privacy (LabelDP), and show that when only the label needs to be protected, the model performance can be significantly improved over the previous state-of-the-art private baselines. Moreover, we study different ways to obtain priors, which when used with RRWithPrior can additionally improve the model performance, further reducing the accuracy gap between private and non-private models. We complement the empirical results with theoretical analysis showing that LabelDP is provably easier than protecting both the inputs and labels.
arXiv commentNeurIPS 2021; 29 pages, 6 figures
References in corpus (25)
- Deep Learning with Differential Privacy
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Improved Regularization of Convolutional Neural Networks with Cutout
- Learning Differentially Private Recurrent Language Models
- Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
- Scalable Private Learning with PATE
- Discrete Distribution Estimation under Local Privacy
- Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
- Normalized Loss Functions for Deep Learning with Noisy Labels
- Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
- Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels
- Private Stochastic Convex Optimization with Optimal Rates
- SIGUA: Forgetting May Make Learning with Noisy Labels More Robust
- Identifying Mislabeled Data using the Area Under the Margin Ranking
- Error-Bounded Correction of Noisy Labels
- Efficient Per-Example Gradient Computations
- Tempered Sigmoid Activations for Deep Learning with Differential Privacy
- Scalable Differential Privacy with Certified Robustness in Adversarial Learning
- Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising
- Individual Privacy Accounting via a Renyi Filter
- Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization
- Leveraging Public Data for Practical Private Query Release
- Auto-Vectorizing TensorFlow Graphs: Jacobians, Auto-Batching And Beyond
- Label differential privacy via clustering
- Differentially Private Regression and Classification with Sparse Gaussian Processes