Weighted Distributed Differential Privacy ERM: Convex and Non-convex

Abstract

Distributed machine learning is an approach allowing different parties to learn a model over all data sets without disclosing their own data. In this paper, we propose a weighted distributed differential privacy (WD-DP) empirical risk minimization (ERM) method to train a model in distributed setting, considering different weights of different clients. We guarantee differential privacy by gradient perturbation, adding Gaussian noise, and advance the state-of-the-art on gradient perturbation method in distributed setting. By detailed theoretical analysis, we show that in distributed setting, the noise bound and the excess empirical risk bound can be improved by considering different weights held by multiple parties. Moreover, considering that the constraint of convex loss function in ERM is not easy to achieve in some situations, we generalize our method to non-convex loss functions which satisfy Polyak-Lojasiewicz condition. Experiments on real data sets show that our method is more reliable and we improve the performance of distributed differential privacy ERM, especially in the case that data scale on different clients is uneven.

References in corpus (16)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Learning Differentially Private Recurrent Language Models
• Differentially Private Empirical Risk Minimization
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Differentially Private Federated Learning: A Client Level Perspective
• DP-ADMM: ADMM-based Distributed Learning with Differential Privacy
• Evaluating Differentially Private Machine Learning in Practice
• Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds
• Functional Mechanism: Regression Analysis under Differential Privacy
• Generalization in Generative Adversarial Networks: A Novel Perspective from Privacy Protection
• Differentially Private Bayesian Linear Regression
• Global Convergence of Arbitrary-Block Gradient Methods for Generalized Polyak-Łojasiewicz Functions
• Efficient Private ERM for Smooth Objectives
• Differentially Private Continual Learning
• Mining Twitter to Assess the Determinants of Health Behavior towards Human Papillomavirus Vaccination in the United States

Cited by in corpus (1)

• Topology-aware Differential Privacy for Decentralized Image Classification