Federated Multi-Task Learning under a Mixture of Distributions

Abstract

The increasing size of data generated by smartphones and IoT devices motivated the development of Federated Learning (FL), a framework for on-device collaborative training of machine learning models. First efforts in FL focused on learning a single global model with good average performance across clients, but the global model may be arbitrarily bad for a given client, due to the inherent heterogeneity of local data distributions. Federated multi-task learning (MTL) approaches can learn personalized models by formulating an opportune penalized optimization problem. The penalization term can capture complex relations among personalized models, but eschews clear statistical assumptions about local data distributions. In this work, we propose to study federated MTL under the flexible assumption that each local data distribution is a mixture of unknown underlying distributions. This assumption encompasses most of the existing personalized FL approaches and leads to federated EM-like algorithms for both client-server and fully decentralized settings. Moreover, it provides a principled way to serve personalized models to clients not seen at training time. The algorithms' convergence is analyzed through a novel federated surrogate optimization framework, which can be of general interest. Experimental results on FL benchmarks show that our approach provides models with higher accuracy and fairness than state-of-the-art methods.

arXiv comment77 pages, NeurIPS 2021

References in corpus (20)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• LEAF: A Benchmark for Federated Settings
• Agnostic Federated Learning
• Federated Learning of a Mixture of Global and Local Models
• D${}^2$: Decentralized Training over Decentralized Data
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Variational Federated Multi-Task Learning
• Optimization with First-Order Surrogate Functions
• Federated Learning with Matched Averaging
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• Personalized Cross-Silo Federated Learning on Non-IID Data
• Personalized Federated Learning using Hypernetworks
• Decentralized gradient methods: does topology matter?

Cited by in corpus (7)

• DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalization
• Intelligent Model Update Strategy for Sequential Recommendation
• Federated brain tumor segmentation: an extensive benchmark
• Personalized Federated Learning through Local Memorization
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
• FedAGHN: Personalized Federated Learning with Attentive Graph HyperNetworks