Efficient Privacy-Preserving Stochastic Nonconvex Optimization

Abstract

While many solutions for privacy-preserving convex empirical risk minimization (ERM) have been developed, privacy-preserving nonconvex ERM remains a challenge. We study nonconvex ERM, which takes the form of minimizing a finite-sum of nonconvex loss functions over a training set. We propose a new differentially private stochastic gradient descent algorithm for nonconvex ERM that achieves strong privacy guarantees efficiently, and provide a tight analysis of its privacy and utility guarantees, as well as its gradient complexity. Our algorithm reduces gradient complexity while improves the best previous utility guarantee given by Wang et al. (NeurIPS 2017). Our experiments on benchmark nonconvex ERM problems demonstrate superior performance in terms of both training cost and utility gains compared with previous differentially private methods using the same privacy budgets.

arXiv comment29 pages, 5 figures, 3 tables. This version corrects a miscalculation in the previous proof, resulting in an improved utility bound for the algorithm

References in corpus (16)

• Deep Learning with Differential Privacy
• Renyi Differential Privacy
• Learning Differentially Private Recurrent Language Models
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Differentially Private Generative Adversarial Network
• SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Non-convex Finite-Sum Optimization Via SCSG Methods
• Gaussian Differential Privacy
• Subsampled Rényi Differential Privacy and Analytical Moments Accountant
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Private Stochastic Convex Optimization with Optimal Rates
• Stochastic Nested Variance Reduction for Nonconvex Optimization
• Differentially Private Bayesian Learning on Distributed Data
• Deep Learning with Gaussian Differential Privacy
• Stochastic Recursive Momentum for Policy Gradient Methods

Cited by in corpus (5)

• Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Data
• Wide Network Learning with Differential Privacy
• Differentially Private Federated Learning with Laplacian Smoothing
• A(DP)${}^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy
• Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds