SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning

Abstract

Federated learning~(FL) facilitates the training and deploying AI models on edge devices. Preserving user data privacy in FL introduces several challenges, including expensive communication costs, limited resources, and data heterogeneity. In this paper, we propose SPATL, an FL method that addresses these issues by: (a) introducing a salient parameter selection agent and communicating selected parameters only; (b) splitting a model into a shared encoder and a local predictor, and transferring its knowledge to heterogeneous clients via the locally customized predictor. Additionally, we leverage a gradient control mechanism to further speed up model convergence and increase robustness of training processes. Experiments demonstrate that SPATL reduces communication overhead, accelerates model inference, and enables stable training processes with better results compared to state-of-the-art methods. Our approach reduces communication cost by up to $86.45\%$, accelerates local inference by reducing up to $39.7\%$ FLOPs on VGG-11, and requires $7.4\times$ less communication overhead when training ResNet-20.

arXiv commentAccepted at The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 22)

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Personalized Federated Learning with Moreau Envelopes
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learning
• Learnable Motion Coherence for Correspondence Pruning
• Auto Graph Encoder-Decoder for Neural Network Pruning