Differentially Private Model Publishing for Deep Learning

Abstract

Deep learning techniques based on neural networks have shown significant success in a wide range of AI tasks. Large-scale training datasets are one of the critical factors for their success. However, when the training datasets are crowdsourced from individuals and contain sensitive information, the model parameters may encode private information and bear the risks of privacy leakage. The recent growing trend of the sharing and publishing of pre-trained models further aggravates such privacy risks. To tackle this problem, we propose a differentially private approach for training neural networks. Our approach includes several new techniques for optimizing both privacy loss and model accuracy. We employ a generalization of differential privacy called concentrated differential privacy(CDP), with both a formal and refined privacy loss analysis on two different data batching methods. We implement a dynamic privacy budget allocator over the course of training to improve model accuracy. Extensive experiments demonstrate that our approach effectively improves privacy loss accounting, training efficiency and model quality under a given privacy budget.

References in corpus (3)

• Deep Learning with Differential Privacy
• Rényi Divergence and Kullback-Leibler Divergence
• Why Random Reshuffling Beats Stochastic Gradient Descent

Cited by in corpus (11)

• DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environments
• Decentralized Federated Learning through Proxy Model Sharing
• Recent Advances of Differential Privacy in Centralized Deep Learning: A Systematic Survey
• Privacy Accounting and Quality Control in the Sage Differentially Private ML Platform
• Learning Privacy-Preserving Student Networks via Discriminative-Generative Distillation
• Implementing Responsible AI: Tensions and Trade-Offs Between Ethics Aspects
• TransMIA: Membership Inference Attacks Using Transfer Shadow Training
• Towards Plausible Differentially Private ADMM Based Distributed Machine Learning
• NeuGuard: Lightweight Neuron-Guided Defense against Membership Inference Attacks
• Privacy Enhancing Machine Learning via Removal of Unwanted Dependencies
• DPack: Efficiency-Oriented Privacy Budget Scheduling