The Skellam Mechanism for Differentially Private Federated Learning

Abstract

We introduce the multi-dimensional Skellam mechanism, a discrete differential privacy mechanism based on the difference of two independent Poisson random variables. To quantify its privacy guarantees, we analyze the privacy loss distribution via a numerical evaluation and provide a sharp bound on the Rényi divergence between two shifted Skellam distributions. While useful in both centralized and distributed privacy applications, we investigate how it can be applied in the context of federated learning with secure aggregation under communication constraints. Our theoretical findings and extensive experimental evaluations demonstrate that the Skellam mechanism provides the same privacy-accuracy trade-offs as the continuous Gaussian mechanism, even when the precision is low. More importantly, Skellam is closed under summation and sampling from it only requires sampling from a Poisson distribution -- an efficient routine that ships with all machine learning and data analysis software packages. These features, along with its discrete nature and competitive privacy-accuracy trade-offs, make it an attractive practical alternative to the newly introduced discrete Gaussian mechanism.

arXiv commentPaper published in NeurIPS 2021

References in corpus (13)

• Deep Learning with Differential Privacy
• Federated Learning: Strategies for Improving Communication Efficiency
• Renyi Differential Privacy
• Federated Learning for Mobile Keyboard Prediction
• Learning Differentially Private Recurrent Language Models
• LEAF: A Benchmark for Federated Settings
• Adaptive Federated Optimization
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Differentially Private Learning with Adaptive Clipping
• Differentially Private Learning Needs Better Features (or Much More Data)
• Practical and Private (Deep) Learning without Sampling or Shuffling
• Training Production Language Models without Memorizing User Data
• Locally Private Bayesian Inference for Count Models

Cited by in corpus (5)

• How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
• The Discrete Gaussian for Differential Privacy
• Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Device Heterogeneity in Federated Learning: A Superquantile Approach