Differentially Private Federated Learning with Laplacian Smoothing

Abstract

Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. However, an adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models. In this paper, we investigate a utility enhancement scheme based on Laplacian smoothing for differentially private federated learning (DP-Fed-LS), where the parameter aggregation with injected Gaussian noise is improved in statistical precision without losing privacy budget. Our key observation is that the aggregated gradients in federated learning often enjoy a type of smoothness, i.e. sparsity in the graph Fourier basis with polynomial decays of Fourier coefficients as frequency grows, which can be exploited by the Laplacian smoothing efficiently. Under a prescribed differential privacy budget, convergence error bounds with tight rates are provided for DP-Fed-LS with uniform subsampling of heterogeneous Non-IID data, revealing possible utility improvement of Laplacian smoothing in effective dimensionality and variance reduction, among others. Experiments over MNIST, SVHN, and Shakespeare datasets show that the proposed method can improve model accuracy with DP-guarantee and membership privacy under both uniform and Poisson subsampling mechanisms.

References in corpus (24)

• Deep Learning with Differential Privacy
• Federated Learning: Strategies for Improving Communication Efficiency
• Renyi Differential Privacy
• Federated Learning for Mobile Keyboard Prediction
• Dota 2 with Large Scale Deep Reinforcement Learning
• On the Convergence of FedAvg on Non-IID Data
• How To Backdoor Federated Learning
• Learning Differentially Private Recurrent Language Models
• Differentially Private Federated Learning: A Client Level Perspective
• Scalable Private Learning with PATE
• LEAF: A Benchmark for Federated Settings
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Distributed Mean Estimation with Limited Communication
• A General Approach to Adding Differential Privacy to Iterative Training Procedures
• Privacy Amplification by Iteration
• White-box vs Black-box: Bayes Optimal Strategies for Membership Inference
• Subsampled Rényi Differential Privacy and Analytical Moments Accountant
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Private Stochastic Convex Optimization with Optimal Rates
• Learning with Differential Privacy: Stability, Learnability and the Sufficiency and Necessity of ERM Principle
• Laplacian Smoothing Gradient Descent
• DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM
• Efficient Privacy-Preserving Stochastic Nonconvex Optimization

Cited by in corpus (4)

• From Distributed Machine Learning to Federated Learning: A Survey
• Do Gradient Inversion Attacks Make Federated Learning Unsafe?
• A Theoretical Perspective on Differentially Private Federated Multi-task Learning
• A Deterministic Gradient-Based Approach to Avoid Saddle Points