Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs

Abstract

We consider a distributed empirical risk minimization (ERM) optimization problem with communication efficiency and privacy requirements, motivated by the federated learning (FL) framework. Unique challenges to the traditional ERM problem in the context of FL include (i) need to provide privacy guarantees on clients' data, (ii) compress the communication between clients and the server, since clients might have low-bandwidth links, (iii) work with a dynamic client population at each round of communication between the server and the clients, as a small fraction of clients are sampled at each round. To address these challenges we develop (optimal) communication-efficient schemes for private mean estimation for several ${\ell }_p$ spaces, enabling efficient gradient aggregation for each iteration of the optimization solution of the ERM. We also provide lower and upper bounds for mean estimation with privacy and communication constraints for arbitrary ${\ell }_p$ spaces. To get the overall communication, privacy, and optimization performance operation point, we combine this with privacy amplification opportunities inherent to this setup. Our solution takes advantage of the inherent privacy amplification provided by client sampling and data sampling at each client (through Stochastic Gradient Descent) as well as the recently developed privacy framework using anonymization, which effectively presents to the server responses that are randomly shuffled with respect to the clients. Putting these together, we demonstrate that one can get the same privacy, optimization-performance operating point developed in recent methods that use full-precision communication, but at a much lower communication cost, i.e., effectively getting communication efficiency for "free".

References in corpus (7)

• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluation
• Communication Complexity in Locally Private Distribution Estimation and Heavy Hitters
• Private Counting from Anonymous Messages: Near-Optimal Accuracy with Vanishing Communication Overhead
• Improved Summation from Shuffling
• SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimization

Cited by in corpus (7)

• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• Differentially Private Federated Learning on Heterogeneous Data
• Lossless Compression of Efficient Private Local Randomizers
• DP-REC: Private & Communication-Efficient Federated Learning
• On Large-Cohort Training for Federated Learning
• QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
• Information-constrained optimization: can adaptive processing of gradients help?