SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling

Abstract

Federated learning enables multiple clients, such as mobile phones and organizations, to collaboratively learn a shared model for prediction while protecting local data privacy. However, most recent research and applications of federated learning assume that all clients have fully labeled data, which is impractical in real-world settings. In this work, we focus on a new scenario for cross-silo federated learning, where data samples of each client are partially labeled. We borrow ideas from semi-supervised learning methods where a large amount of unlabeled data is utilized to improve the model's accuracy despite limited access to labeled examples. We propose a new framework dubbed SemiFed that unifies two dominant approaches for semi-supervised learning: consistency regularization and pseudo-labeling. SemiFed first applies advanced data augmentation techniques to enforce consistency regularization and then generates pseudo-labels using the model's predictions during training. SemiFed takes advantage of the federation so that for a given image, the pseudo-label holds only if multiple models from different clients produce a high-confidence prediction and agree on the same label. Extensive experiments on two image benchmarks demonstrate the effectiveness of our approach under both homogeneous and heterogeneous data distribution settings

References in corpus (15)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• AutoAugment: Learning Augmentation Policies from Data
• Learning with Pseudo-Ensembles
• FedML: A Research Library and Benchmark for Federated Machine Learning
• A generic framework for privacy preserving deep learning
• LEAF: A Benchmark for Federated Settings
• Bayesian Nonparametric Federated Learning of Neural Networks
• One-Shot Federated Learning

Cited by in corpus (2)

• Uncertainty Minimization for Personalized Federated Semi-Supervised Learning
• Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling