The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation

Abstract

We consider training models on private data that are distributed across user devices. To ensure privacy, we add on-device noise and use secure aggregation so that only the noisy sum is revealed to the server. We present a comprehensive end-to-end system, which appropriately discretizes the data and adds discrete Gaussian noise before performing secure aggregation. We provide a novel privacy analysis for sums of discrete Gaussians and carefully analyze the effects of data quantization and modular summation arithmetic. Our theoretical guarantees highlight the complex tension between communication, privacy, and accuracy. Our extensive experimental results demonstrate that our solution is essentially able to match the accuracy to central differential privacy with less than 16 bits of precision per value.

arXiv commentInternational Conference on Machine Learning (ICML), 2021

References in corpus (22)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Renyi Differential Privacy
• Federated Learning for Mobile Keyboard Prediction
• Towards Federated Learning at Scale: System Design
• Learning Differentially Private Recurrent Language Models
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• LEAF: A Benchmark for Federated Settings
• Discrete Distribution Estimation under Local Privacy
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Distributed Mean Estimation with Limited Communication
• Adaptive Federated Optimization
• Subsampled Rényi Differential Privacy and Analytical Moments Accountant
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Differentially Private Learning with Adaptive Clipping
• Differentially Private Learning Needs Better Features (or Much More Data)
• Hypothesis Testing Interpretations and Renyi Differential Privacy
• On the Power of Multiple Anonymous Messages
• Practical and Private (Deep) Learning without Sampling or Shuffling
• D2P-Fed: Differentially Private Federated Learning With Efficient Communication
• Private Counting from Anonymous Messages: Near-Optimal Accuracy with Vanishing Communication Overhead
• Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs

Cited by in corpus (9)

• DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation
• The Discrete Gaussian for Differential Privacy
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Communication-Efficient Agnostic Federated Averaging
• DP-REC: Private & Communication-Efficient Federated Learning
• D3p -- A Python Package for Differentially-Private Probabilistic Programming
• FED-${\chi }^2$: Privacy Preserving Federated Correlation Test
• Locally Private k-Means in One Round