Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising

Abstract

Deep learning models leak significant amounts of information about their training datasets. Previous work has investigated training models with differential privacy (DP) guarantees through adding DP noise to the gradients. However, such solutions (specifically, DPSGD), result in large degradations in the accuracy of the trained models. In this paper, we aim at training deep learning models with DP guarantees while preserving model accuracy much better than previous works. Our key technique is to encode gradients to map them to a smaller vector space, therefore enabling us to obtain DP guarantees for different noise distributions. This allows us to investigate and choose noise distributions that best preserve model accuracy for a target privacy budget. We also take advantage of the post-processing property of differential privacy by introducing the idea of denoising, which further improves the utility of the trained models without degrading their DP guarantees. We show that our mechanism outperforms the state-of-the-art DPSGD; for instance, for the same model accuracy of $96.1\%$ on MNIST, our technique results in a privacy bound of $\epsilon =3.2$ compared to $\epsilon =6$ of DPSGD, which is a significant improvement.

References in corpus (6)

• Deep Learning with Differential Privacy
• Renyi Differential Privacy
• Learning Differentially Private Recurrent Language Models
• Privacy Amplification by Iteration
• Subsampled Rényi Differential Privacy and Analytical Moments Accountant
• Rényi Differential Privacy of the Sampled Gaussian Mechanism

Cited by in corpus (8)

• Differentially Private Learning Needs Better Features (or Much More Data)
• Learning Privacy-Preserving Student Networks via Discriminative-Generative Distillation
• Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture
• DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling
• Differentially Private Federated Learning for Cancer Prediction
• Deep Learning with Label Differential Privacy
• Improving the Algorithm of Deep Learning with Differential Privacy
• Generative Models for Security: Attacks, Defenses, and Opportunities