Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence

Abstract

Although machine learning models trained on massive data have led to break-throughs in several areas, their deployment in privacy-sensitive domains remains limited due to restricted access to data. Generative models trained with privacy constraints on private data can sidestep this challenge, providing indirect access to private data instead. We propose DP-Sinkhorn, a novel optimal transport-based generative method for learning data distributions from private data with differential privacy. DP-Sinkhorn minimizes the Sinkhorn divergence, a computationally efficient approximation to the exact optimal transport distance, between the model and data in a differentially private manner and uses a novel technique for control-ling the bias-variance trade-off of gradient estimates. Unlike existing approaches for training differentially private generative models, which are mostly based on generative adversarial networks, we do not rely on adversarial objectives, which are notoriously difficult to optimize, especially in the presence of noise imposed by privacy constraints. Hence, DP-Sinkhorn is easy to train and deploy. Experimentally, we improve upon the state-of-the-art on multiple image modeling benchmarks and show differentially private synthesis of informative RGB images. Project page:https://nv-tlabs.github.io/DP-Sinkhorn.

arXiv commentAccepted to NeurIPS 2021. 13 pages, 7 pages of supplementary; 6 tables, 8 figures

References in corpus (24)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Learning with Differential Privacy
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Renyi Differential Privacy
• Deep Face Recognition: A Survey
• Which Training Methods for GANs do actually Converge?
• Differentially Private Generative Adversarial Network
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• The Numerics of GANs
• Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences
• On Variational Bounds of Mutual Information
• Interpolating between Optimal Transport and MMD using Sinkhorn Divergences
• From optimal transport to generative modeling: the VEGAN cookbook
• Subsampled Rényi Differential Privacy and Analytical Moments Accountant
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation
• DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generation
• This Person (Probably) Exists. Identity Membership Attacks Against GAN Generated Faces
• Self-Diagnosing GAN: Diagnosing Underrepresented Samples in Generative Adversarial Networks
• Fair Decision Making using Privacy-Protected Data
• P3GM: Private High-Dimensional Data Release via Privacy Preserving Phased Generative Model