FedSkel: Efficient Federated Learning on Heterogeneous Systems with Skeleton Gradients Update

Abstract

Federated learning aims to protect users' privacy while performing data analysis from different participants. However, it is challenging to guarantee the training efficiency on heterogeneous systems due to the various computational capabilities and communication bottlenecks. In this work, we propose FedSkel to enable computation-efficient and communication-efficient federated learning on edge devices by only updating the model's essential parts, named skeleton networks. FedSkel is evaluated on real edge devices with imbalanced datasets. Experimental results show that it could achieve up to 5.52$\times$ speedups for CONV layers' back-propagation, 1.82$\times$ speedups for the whole training process, and reduce 64.8% communication cost, with negligible accuracy loss.

arXiv commentCIKM 2021

References in corpus (12)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Federated Learning: Strategies for Improving Communication Efficiency
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• Federated Learning for Emoji Prediction in a Mobile Keyboard
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Federated Multi-Task Learning
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
• Distilling Critical Paths in Convolutional Neural Networks

Cited by in corpus (1)

• Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models