Decentralized Differentially Private Without-Replacement Stochastic Gradient Descent

Abstract

While machine learning has achieved remarkable results in a wide variety of domains, the training of models often requires large datasets that may need to be collected from different individuals. As sensitive information may be contained in the individual's dataset, sharing training data may lead to severe privacy concerns. Therefore, there is a compelling need to develop privacy-aware machine learning methods, for which one effective approach is to leverage the generic framework of differential privacy. Considering that stochastic gradient descent (SGD) is one of the most commonly adopted methods for large-scale machine learning problems, a decentralized differentially private SGD algorithm is proposed in this work. Particularly, we focus on SGD without replacement due to its favorable structure for practical implementation. Both privacy and convergence analysis are provided for the proposed algorithm. Finally, extensive experiments are performed to demonstrate the effectiveness of the proposed method.

References in corpus (12)

• Deep Learning with Differential Privacy
• Optimal Distributed Online Prediction using Mini-Batches
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Train faster, generalize better: Stability of stochastic gradient descent
• Scalable Private Learning with PATE
• cpSGD: Communication-efficient and differentially-private distributed SGD
• Collecting and Analyzing Data from Smart Device Users with Local Differential Privacy
• Privacy Amplification by Iteration
• Why Random Reshuffling Beats Stochastic Gradient Descent
• Enabling large-scale viscoelastic calculations via neural network acceleration
• Accuracy First: Selecting a Differential Privacy Level for Accuracy-Constrained ERM
• Without-Replacement Sampling for Stochastic Gradient Methods: Convergence Results and Application to Distributed Optimization