Personalized Federated Learning with First Order Model Optimization

Abstract

While federated learning traditionally aims to train a single global model across decentralized local datasets, one model may not always be ideal for all participating clients. Here we propose an alternative, where each client only federates with other relevant clients to obtain a stronger model per client-specific objectives. To achieve this personalization, rather than computing a single model average with constant weights for the entire federation as in traditional FL, we efficiently calculate optimal weighted model combinations for each client, based on figuring out how much a client can benefit from another's model. We do not assume knowledge of any underlying data distributions or client similarities, and allow each client to optimize for arbitrary target distributions of interest, enabling greater flexibility for personalization. We evaluate and characterize our method on a variety of federated settings, datasets, and degrees of local data heterogeneity. Our method outperforms existing alternatives, while also enabling new features for personalized FL such as transfer outside of local data distributions.

ICLR 2021

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Towards Federated Learning at Scale: System Design
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Federated Learning of a Mixture of Global and Local Models
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Private Federated Learning with Domain Adaptation

Cited by in corpus (5)

• FedALA: Adaptive Local Aggregation for Personalized Federated Learning
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• Personalized Federated Learning using Hypernetworks
• Personalized Federated Learning with Gaussian Processes
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks