Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models

Abstract

Federated learning (FL) is a promising way to use the computing power of mobile devices while maintaining the privacy of users. Current work in FL, however, makes the unrealistic assumption that the users have ground-truth labels on their devices, while also assuming that the server has neither data nor labels. In this work, we consider the more realistic scenario where the users have only unlabeled data, while the server has some labeled data, and where the amount of labeled data is smaller than the amount of unlabeled data. We call this learning problem semi-supervised federated learning (SSFL). For SSFL, we demonstrate that a critical issue that affects the test accuracy is the large gradient diversity of the models from different users. Based on this, we investigate several design choices. First, we find that the so-called consistency regularization loss (CRL), which is widely used in semi-supervised learning, performs reasonably well but has large gradient diversity. Second, we find that Batch Normalization (BN) increases gradient diversity. Replacing BN with the recently-proposed Group Normalization (GN) can reduce gradient diversity and improve test accuracy. Third, we show that CRL combined with GN still has a large gradient diversity when the number of users is large. Based on these results, we propose a novel grouping-based model averaging method to replace the FedAvg averaging method. Overall, our grouping-based averaging, combined with GN and CRL, achieves better test accuracy than not just a contemporary paper on SSFL in the same settings (>10\%), but also four supervised FL algorithms.

References in corpus (25)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• mixup: Beyond Empirical Risk Minimization
• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Federated Learning with Non-IID Data
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Unsupervised Domain Adaptation with Residual Transfer Networks
• AutoAugment: Learning Augmentation Policies from Data
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Semi-Supervised Learning with Ladder Networks
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
• Three Approaches for Personalization with Applications to Federated Learning
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Federated Adversarial Domain Adaptation
• One-Shot Federated Learning
• Adaptive Federated Optimization
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• Semi-supervised Federated Learning for Activity Recognition

Cited by in corpus (5)

• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• Federated Learning on Non-IID Data: A Survey
• FedSEAL: Semi-Supervised Federated Learning with Self-Ensemble Learning and Negative Learning
• Adversarial Robustness through Bias Variance Decomposition: A New Perspective for Federated Learning