SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision

Abstract

Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices in order to strengthen data privacy. An essential but rarely studied challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL compared to centralized training due to the fact that FL users are often reluctant to label their private data. Furthermore, due to the heterogeneous nature of the data at edge devices, it is crucial to develop personalized models. In this paper we propose self-supervised federated learning (SSFL), a unified self-supervised and personalized federated learning framework, and a series of algorithms under this framework which work towards addressing these challenges. First, under the SSFL framework, we demonstrate that the standard FedAvg algorithm is compatible with recent breakthroughs in centralized self-supervised learning such as SimSiam networks. Moreover, to deal with data heterogeneity at the edge devices in this framework, we have innovated a series of algorithms that broaden existing supervised personalization algorithms into the setting of self-supervised learning. We further propose a novel personalized federated self-supervised learning algorithm, Per-SSFL, which balances personalization and consensus by carefully regulating the distance between the local and global representations of data. To provide a comprehensive comparative analysis of all proposed algorithms, we also develop a distributed training system and related evaluation protocol for SSFL. Our findings show that the gap of evaluation accuracy between supervised learning and unsupervised learning in FL is both small and reasonable. The performance comparison indicates the representation regularization-based personalization method is able to outperform other variants.

References in corpus (19)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Federated Optimization in Heterogeneous Networks
• Exploring Simple Siamese Representation Learning
• Personalized Federated Learning: A Meta-Learning Approach
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• A Field Guide to Federated Optimization
• Adaptive Federated Optimization
• Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances
• Semi-supervised Federated Learning for Activity Recognition
• Self-supervised Cross-silo Federated Neural Architecture Search
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models

Cited by in corpus (1)

• FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks