Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacy

Abstract

Federated learning (FL) enables distributed clients to collaboratively train a machine learning model without sharing raw data with each other. However, it suffers the leakage of private information from uploading models. In addition, as the model size grows, the training latency increases due to limited transmission bandwidth and the model performance degrades while using differential privacy (DP) protection. In this paper, we propose a gradient sparsification empowered FL framework over wireless channels, in order to improve training efficiency without sacrificing convergence performance. Specifically, we first design a random sparsification algorithm to retain a fraction of the gradient elements in each client's local training, thereby mitigating the performance degradation induced by DP and and reducing the number of transmission parameters over wireless channels. Then, we analyze the convergence bound of the proposed algorithm, by modeling a non-convex FL problem. Next, we formulate a time-sequential stochastic optimization problem for minimizing the developed convergence bound, under the constraints of transmit power, the average transmitting delay, as well as the client's DP requirement. Utilizing the Lyapunov drift-plus-penalty framework, we develop an analytical solution to the optimization problem. Extensive experiments have been implemented on three real life datasets to demonstrate the effectiveness of our proposed algorithm. We show that our proposed algorithms can fully exploit the interworking between communication and computation to outperform the baselines, i.e., random scheduling, round robin and delay-minimization algorithms.

References in corpus (13)

• Deep Learning with Differential Privacy
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation
• Multi-Armed Bandit Based Client Scheduling for Federated Learning
• An Efficiency-boosting Client Selection Scheme for Federated Learning with Fairness Guarantee
• Enabling AI in Future Wireless Networks: A Data Life Cycle Perspective
• Bayesian Nonparametric Federated Learning of Neural Networks
• Quantized Federated Learning under Transmission Delay and Outage Constraints
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Semi-Synchronous Personalized Federated Learning over Mobile Edge Networks
• Opacus: User-Friendly Differential Privacy Library in PyTorch
• Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification

Cited by in corpus (1)

• Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization