DP-REC: Private & Communication-Efficient Federated Learning
arXiv:2111.05454
Abstract
Privacy and communication efficiency are important challenges in federated training of neural networks, and combining them is still an open problem. In this work, we develop a method that unifies highly compressed communication and differential privacy (DP). We introduce a compression technique based on Relative Entropy Coding (REC) to the federated setting. With a minor modification to REC, we obtain a provably differentially private learning algorithm, DP-REC, and show how to compute its privacy guarantees. Our experiments demonstrate that DP-REC drastically reduces communication costs while providing privacy guarantees comparable to the state-of-the-art.
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Deep Learning with Differential Privacy
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Scaling Laws for Neural Language Models
- Renyi Differential Privacy
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- LEAF: A Benchmark for Federated Settings
- A Field Guide to Federated Optimization
- Scaling Laws for Autoregressive Generative Modeling
- Adaptive Federated Optimization
- Rényi Differential Privacy of the Sampled Gaussian Mechanism
- Differentially Private Learning with Adaptive Clipping
- Characterising Bias in Compressed Models
- The Discrete Gaussian for Differential Privacy
- Differentially-Private "Draw and Discard" Machine Learning
- The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
- Understanding Unintended Memorization in Federated Learning
- Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs
- Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters