Personalized Federated Learning with Gaussian Processes

Abstract

Federated learning aims to learn a global model that performs well on client devices with limited cross-client communication. Personalized federated learning (PFL) further extends this setup to handle data heterogeneity between clients by learning personalized models. A key challenge in this setting is to learn effectively across clients even though each client has unique data that is often limited in size. Here we present pFedGP, a solution to PFL that is based on Gaussian processes (GPs) with deep kernel learning. GPs are highly expressive models that work well in the low data regime due to their Bayesian nature. However, applying GPs to PFL raises multiple challenges. Mainly, GPs performance depends heavily on access to a good kernel function, and learning a kernel requires a large training set. Therefore, we propose learning a shared kernel function across all clients, parameterized by a neural network, with a personal GP classifier for each client. We further extend pFedGP to include inducing points using two novel methods, the first helps to improve generalization in the low data regime and the second reduces the computational cost. We derive a PAC-Bayes generalization bound on novel clients and empirically show that it gives non-vacuous guarantees. Extensive experiments on standard PFL benchmarks with CIFAR-10, CIFAR-100, and CINIC-10, and on a new setup of learning under input noise show that pFedGP achieves well-calibrated predictions while significantly outperforming baseline methods, reaching up to 21% in accuracy gain.

References in corpus (28)

• Prototypical Networks for Few-shot Learning
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Learning Differentially Private Recurrent Language Models
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Federated Learning with Personalization Layers
• Federated Optimization in Heterogeneous Networks
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Scalable Variational Gaussian Process Classification
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Federated Learning of a Mixture of Global and Local Models
• On the Convergence of Local Descent Methods in Federated Learning
• Bayesian Nonparametric Federated Learning of Neural Networks
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Variational Federated Multi-Task Learning
• Stochastic Variational Deep Kernel Learning
• Federated Learning with Matched Averaging
• Dependent Multinomial Models Made Easy: Stick Breaking with the Pólya-Gamma Augmentation
• Alpha MAML: Adaptive Model-Agnostic Meta-Learning
• Personalized Cross-Silo Federated Learning on Non-IID Data
• Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning
• Personalized Federated Learning with First Order Model Optimization
• Learning Gaussian Processes by Minimizing PAC-Bayesian Generalization Bounds
• Partitioned Variational Inference: A unified framework encompassing federated and continual learning
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?