Differential Privacy Meets Federated Learning under Communication Constraints
arXiv:2101.12240
Abstract
The performance of federated learning systems is bottlenecked by communication costs and training variance. The communication overhead problem is usually addressed by three communication-reduction techniques, namely, model compression, partial device participation, and periodic aggregation, at the cost of increased training variance. Different from traditional distributed learning systems, federated learning suffers from data heterogeneity (since the devices sample their data from possibly different distributions), which induces additional variance among devices during training. Various variance-reduced training algorithms have been introduced to combat the effects of data heterogeneity, while they usually cost additional communication resources to deliver necessary control information. Additionally, data privacy remains a critical issue in FL, and thus there have been attempts at bringing Differential Privacy to this framework as a mediator between utility and privacy requirements. This paper investigates the trade-offs between communication costs and training variance under a resource-constrained federated system theoretically and experimentally, and how communication reduction techniques interplay in a differentially private setting. The results provide important insights into designing practical privacy-aware federated learning systems.
arXiv comment11 pages, 4 figures
References in corpus (11)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Revisiting Distributed Synchronous SGD
- Federated Optimization in Heterogeneous Networks
- Communication Efficient Distributed Optimization using an Approximate Newton-type Method
- Subsampled Rényi Differential Privacy and Analytical Moments Accountant
- First Analysis of Local GD on Heterogeneous Data
- The Discrete Gaussian for Differential Privacy
- LDP-FL: Practical Private Aggregation in Federated Learning with Local Differential Privacy
- D2P-Fed: Differentially Private Federated Learning With Efficient Communication