Learning Privacy-Preserving Student Networks via Discriminative-Generative Distillation

Abstract

While deep models have proved successful in learning rich knowledge from massive well-annotated data, they may pose a privacy leakage risk in practical deployment. It is necessary to find an effective trade-off between high utility and strong privacy. In this work, we propose a discriminative-generative distillation approach to learn privacy-preserving deep models. Our key idea is taking models as bridge to distill knowledge from private data and then transfer it to learn a student network via two streams. First, discriminative stream trains a baseline classifier on private data and an ensemble of teachers on multiple disjoint private subsets, respectively. Then, generative stream takes the classifier as a fixed discriminator and trains a generator in a data-free manner. After that, the generator is used to generate massive synthetic data which are further applied to train a variational autoencoder (VAE). Among these synthetic data, a few of them are fed into the teacher ensemble to query labels via differentially private aggregation, while most of them are embedded to the trained VAE for reconstructing synthetic data. Finally, a semi-supervised student learning is performed to simultaneously handle two tasks: knowledge transfer from the teachers with distillation on few privately labeled synthetic data, and knowledge enhancement with tangent-normal adversarial regularization on many triples of reconstructed synthetic data. In this way, our approach can control query cost over private data and mitigate accuracy degradation in a unified manner, leading to a privacy-preserving student model. Extensive experiments and analysis clearly show the effectiveness of the proposed approach.

arXiv commentThis paper is accepted by IEEE Transactions on Image Processing (TIP)

References in corpus (24)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• Deep Learning with Differential Privacy
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Knowledge Distillation: A Survey
• FitNets: Hints for Thin Deep Nets
• RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response
• Semi-Supervised Learning with Generative Adversarial Networks
• Scalable Private Learning with PATE
• Differentially Private Generative Adversarial Network
• Good Semi-supervised Learning that Requires a Bad GAN
• Differentially Private Model Publishing for Deep Learning
• Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition
• Double Similarity Distillation for Semantic Image Segmentation
• Understanding Gradient Clipping in Private SGD: A Geometric Perspective
• DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation
• Rain O'er Me: Synthesizing real rain to derain with data distillation
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
• DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generation
• Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising
• G-PATE: Scalable Differentially Private Data Generator via Private Aggregation of Teacher Discriminators
• Large Scale Private Learning via Low-rank Reparametrization