Private Deep Learning with Teacher Ensembles

Abstract

Privacy-preserving deep learning is crucial for deploying deep neural network based solutions, especially when the model works on data that contains sensitive information. Most privacy-preserving methods lead to undesirable performance degradation. Ensemble learning is an effective way to improve model performance. In this work, we propose a new method for teacher ensembles that uses more informative network outputs under differential private stochastic gradient descent and provide provable privacy guarantees. Out method employs knowledge distillation and hint learning on intermediate representations to facilitate the training of student model. Additionally, we propose a simple weighted ensemble scheme that works more robustly across different teaching settings. Experimental results on three common image datasets benchmark (i.e., CIFAR10, MINST, and SVHN) demonstrate that our approach outperforms previous state-of-the-art methods on both performance and privacy-budget.

arXiv commentfixed updated version will be updated later

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Deep Learning with Differential Privacy
• Learning Differentially Private Recurrent Language Models
• Scalable Private Learning with PATE
• Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models
• Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds
• Private Model Compression via Knowledge Distillation