Connecting Low-Loss Subspace for Personalized Federated Learning
arXiv:2109.07628 · doi:10.1145/3534678.3539254
Abstract
Due to the curse of statistical heterogeneity across clients, adopting a personalized federated learning method has become an essential choice for the successful deployment of federated learning-based services. Among diverse branches of personalization techniques, a model mixture-based personalization method is preferred as each client has their own personalized model as a result of federated learning. It usually requires a local model and a federated model, but this approach is either limited to partial parameter exchange or requires additional local updates, each of which is helpless to novel clients and burdensome to the client's computational capacity. As the existence of a connected subspace containing diverse low-loss solutions between two or more independent deep networks has been discovered, we combined this interesting property with the model mixture-based personalized federated learning method for improved performance of personalization. We proposed SuPerFed, a personalized federated learning method that induces an explicit connection between the optima of the local and the federated model in weight space for boosting each other. Through extensive experiments on several benchmark datasets, we demonstrated that our method achieves consistent gains in both personalization performance and robustness to problematic scenarios possible in realistic services.
arXiv commentAppears at ACM SIGKDD 2022. Code available at http://github.com/vaseline555/SuPerFed
References in corpus (25)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Federated Learning with Non-IID Data
- On Calibration of Modern Neural Networks
- Temporal Ensembling for Semi-Supervised Learning
- Robust Aggregation for Federated Learning
- The Loss Surfaces of Multilayer Networks
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- Federated Learning with Personalization Layers
- Improving Federated Learning Personalization via Model Agnostic Meta Learning
- Deep Ensembles: A Loss Landscape Perspective
- FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
- Adaptive Personalized Federated Learning
- Three Approaches for Personalization with Applications to Federated Learning
- Federated Learning of a Mixture of Global and Local Models
- Ditto: Fair and Robust Federated Learning Through Personalization
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Qualitatively characterizing neural network optimization problems
- Essentially No Barriers in Neural Network Energy Landscape
- A Simple Baseline for Bayesian Uncertainty in Deep Learning
- Multivariate Confidence Calibration for Object Detection
- Linear Mode Connectivity and the Lottery Ticket Hypothesis
- Federated Multi-Task Learning under a Mixture of Distributions
- Large Scale Structure of Neural Network Loss Landscapes
- Learning Neural Network Subspaces
- Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"