FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
arXiv:2110.11027
Abstract
Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints. Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes. In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL. By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance. Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server on various image classification tasks.
arXiv commentUnder review as a conference paper at ICLR 2022
References in corpus (18)
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Language Models are Few-Shot Learners
- Towards Federated Learning at Scale: System Design
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- Ensemble Distillation for Robust Model Fusion in Federated Learning
- FedMD: Heterogenous Federated Learning via Model Distillation
- FedML: A Research Library and Benchmark for Federated Machine Learning
- Fair Resource Allocation in Federated Learning
- Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
- LEAF: A Benchmark for Federated Settings
- Don't Use Large Mini-Batches, Use Local SGD
- Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
- Federated Multi-Task Learning
- Abnormal Client Behavior Detection in Federated Learning
- Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
- NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
- Reinforced Multi-Teacher Selection for Knowledge Distillation