Wide Network Learning with Differential Privacy

Abstract

Despite intense interest and considerable effort, the current generation of neural networks suffers a significant loss of accuracy under most practically relevant privacy training regimes. One particularly challenging class of neural networks are the wide ones, such as those deployed for NLP typeahead prediction or recommender systems. Observing that these models share something in common--an embedding layer that reduces the dimensionality of the input--we focus on developing a general approach towards training these models that takes advantage of the sparsity of the gradients. More abstractly, we address the problem of differentially private empirical risk minimization (ERM) for models that admit sparse gradients. We demonstrate that for non-convex ERM problems, the loss is logarithmically dependent on the number of parameters, in contrast with polynomial dependence for the general case. Following the same intuition, we propose a novel algorithm for privately training neural networks. Finally, we provide an empirical study of a DP wide neural network on a real-world dataset, which has been rarely explored in the previous work.

References in corpus (17)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Deep Learning with Differential Privacy
• Learning Differentially Private Recurrent Language Models
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Scalable Private Learning with PATE
• Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences
• Subsampled Rényi Differential Privacy and Analytical Moments Accountant
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Differentially Private Learning with Adaptive Clipping
• The Cost of Privacy: Rates of Convergence for Parameter Estimation with Differential Privacy
• Differentially Private Data Generative Models
• Training Production Language Models without Memorizing User Data
• Efficient Privacy-Preserving Stochastic Nonconvex Optimization
• dpUGC: Learn Differentially Private Representation for User Generated Contents
• Differentially Private Deep Learning with Direct Feedback Alignment
• A Simple Deep Personalized Recommendation System

Cited by in corpus (6)

• Large Language Models Can Be Strong Differentially Private Learners
• Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification
• Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Data
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• Statistical Inference in the Differential Privacy Model
• Privacy accounting $\epsilon$conomics: Improving differential privacy composition via a posteriori bounds