Personalized Federated Learning using Hypernetworks

Abstract

Personalized federated learning is tasked with training machine learning models for multiple clients, each with its own data distribution. The goal is to train personalized models in a collaborative way while accounting for data disparities across clients and reducing communication costs. We propose a novel approach to this problem using hypernetworks, termed pFedHN for personalized Federated HyperNetworks. In this approach, a central hypernetwork model is trained to generate a set of models, one model for each client. This architecture provides effective parameter sharing across clients, while maintaining the capacity to generate unique and diverse personal models. Furthermore, since hypernetwork parameters are never transmitted, this approach decouples the communication cost from the trainable model size. We test pFedHN empirically in several personalized federated learning challenges and find that it outperforms previous methods. Finally, since hypernetworks share information across clients we show that pFedHN can generalize better to new clients whose distributions differ from any client observed during training.

References in corpus (23)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Federated Optimization in Heterogeneous Networks
• Lookahead Optimizer: k steps forward, 1 step back
• Personalized Federated Learning: A Meta-Learning Approach
• Three Approaches for Personalization with Applications to Federated Learning
• Personalized Federated Learning with Moreau Envelopes
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• On the Convergence of Local Descent Methods in Federated Learning
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Federated Multi-Task Learning
• Stochastic Hyperparameter Optimization through Hypernetworks
• Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
• Alpha MAML: Adaptive Model-Agnostic Meta-Learning
• Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning
• Personalized Federated Learning with First Order Model Optimization
• Hyper-Graph-Network Decoders for Block Codes
• Federated Heavy Hitters Discovery with Differential Privacy
• Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians

Cited by in corpus (6)

• FedALA: Adaptive Local Aggregation for Personalized Federated Learning
• Federated Multi-Task Learning under a Mixture of Distributions
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• Learning the Pareto Front with Hypernetworks
• Personalized Federated Learning through Local Memorization
• Towards Fleet-wide Sharing of Wind Turbine Condition Information through Privacy-preserving Federated Learning