FedSKETCH: Communication-Efficient and Private Federated Learning via Sketching
arXiv:2008.04975
Abstract
Communication complexity and privacy are the two key challenges in Federated Learning where the goal is to perform a distributed learning through a large volume of devices. In this work, we introduce FedSKETCH and FedSKETCHGATE algorithms to address both challenges in Federated learning jointly, where these algorithms are intended to be used for homogeneous and heterogeneous data distribution settings respectively. The key idea is to compress the accumulation of local gradients using count sketch, therefore, the server does not have access to the gradients themselves which provides privacy. Furthermore, due to the lower dimension of sketching used, our method exhibits communication-efficiency property as well. We provide, for the aforementioned schemes, sharp convergence guarantees. Finally, we back up our theory with various set of experiments.
References in corpus (27)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Learning Differentially Private Recurrent Language Models
- SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
- Differentially Private Federated Learning: A Client Level Perspective
- Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
- Sparsified SGD with Memory
- Communication Compression for Decentralized Training
- On the Convergence of Local Descent Methods in Federated Learning
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
- Adaptive Federated Optimization
- Variance Reduced Local SGD with Lower Communication Complexity
- Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
- Parallel SGD: When does averaging help?
- The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
- FetchSGD: Communication-Efficient Federated Learning with Sketching
- Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
- Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
- Federated Learning with Compression: Unified Analysis and Sharp Guarantees
- Enhancing the Privacy of Federated Learning with Sketching
- A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
- Total stochastic gradient algorithms and applications in reinforcement learning
- Gradient Distribution Priors for Biomedical Image Processing