SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
arXiv:2106.01432
Abstract
Federated Learning allows the training of machine learning models by using the computation and private data resources of many distributed clients. Most existing results on Federated Learning (FL) assume the clients have ground-truth labels. However, in many practical scenarios, clients may be unable to label task-specific data due to a lack of expertise or resource. We propose SemiFL to address the problem of combining communication-efficient FL such as FedAvg with Semi-Supervised Learning (SSL). In SemiFL, clients have completely unlabeled data and can train multiple local epochs to reduce communication costs, while the server has a small amount of labeled data. We provide a theoretical understanding of the success of data augmentation-based SSL methods to illustrate the bottleneck of a vanilla combination of communication-efficient FL with SSL. To address this issue, we propose alternate training to `fine-tune global model with labeled data' and `generate pseudo-labels with the global model.' We conduct extensive experiments and demonstrate that our approach significantly improves the performance of a labeled server with unlabeled clients training with multiple local epochs. Moreover, our method outperforms many existing SSFL baselines and performs competitively with the state-of-the-art FL and SSL results.
References in corpus (39)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- mixup: Beyond Empirical Risk Minimization
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Federated Learning: Strategies for Improving Communication Efficiency
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Wide Residual Networks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Unsupervised Data Augmentation for Consistency Training
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Temporal Ensembling for Semi-Supervised Learning
- Federated Learning for Mobile Keyboard Prediction
- Towards Federated Learning at Scale: System Design
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- MixMatch: A Holistic Approach to Semi-Supervised Learning
- FedMD: Heterogenous Federated Learning via Model Distillation
- Federated Optimization in Heterogeneous Networks
- Improving Federated Learning Personalization via Model Agnostic Meta Learning
- FedML: A Research Library and Benchmark for Federated Machine Learning
- Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
- FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
- A Simple Semi-Supervised Learning Framework for Object Detection
- Three Approaches for Personalization with Applications to Federated Learning
- Good Semi-supervised Learning that Requires a Bad GAN
- HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
- ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
- Federated Learning Based on Dynamic Regularization
- Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
- LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
- SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
- FedMix: Approximation of Mixup under Mean Augmented Federated Learning
- Self-ensembling for visual domain adaptation
- Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
- Semi-supervised Federated Learning for Activity Recognition
- Fast learning rates for plug-in classifiers under the margin condition
- Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
- Communication-Efficient Federated Distillation
- Exploiting Unlabeled Data in Smart Cities using Federated Learning