The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation

Abstract

We consider training models on private data that are distributed across user devices. To ensure privacy, we add on-device noise and use secure aggregation so that only the noisy sum is revealed to the server. We present a comprehensive end-to-end system, which appropriately discretizes the data and adds discrete Gaussian noise before performing secure aggregation. We provide a novel privacy analysis for sums of discrete Gaussians and carefully analyze the effects of data quantization and modular summation arithmetic. Our theoretical guarantees highlight the complex tension between communication, privacy, and accuracy. Our extensive experimental results demonstrate that our solution is essentially able to match the accuracy to central differential privacy with less than 16 bits of precision per value.

International Conference on Machine Learning (ICML), 2021

References in corpus (6)

• Towards Federated Learning at Scale: System Design
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Differentially Private Learning Needs Better Features (or Much More Data)
• Practical and Private (Deep) Learning without Sampling or Shuffling
• Private Counting from Anonymous Messages: Near-Optimal Accuracy with Vanishing Communication Overhead
• Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs

Cited by in corpus (7)

• DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• DP-REC: Private & Communication-Efficient Federated Learning
• Communication-Efficient Agnostic Federated Averaging
• FED-${\chi }^2$: Privacy Preserving Federated Correlation Test
• D3p -- A Python Package for Differentially-Private Probabilistic Programming
• Locally Private k-Means in One Round