Three Approaches for Personalization with Applications to Federated Learning

Abstract

The standard objective in machine learning is to train a single model for all users. However, in many learning scenarios, such as cloud computing and federated learning, it is possible to learn a personalized model per user. In this work, we present a systematic learning-theoretic study of personalization. We propose and analyze three approaches: user clustering, data interpolation, and model interpolation. For all three approaches, we provide learning-theoretic guarantees and efficient algorithms for which we also demonstrate the performance empirically. All of our algorithms are model-agnostic and work for any hypothesis class.

24 pages

References in corpus (31)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Federated Learning for Mobile Keyboard Prediction
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Federated Learning with Personalization Layers
• Domain Adaptation: Learning Bounds and Algorithms
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• Adaptive Personalized Federated Learning
• LEAF: A Benchmark for Federated Settings
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Agnostic Federated Learning
• Federated Learning of a Mixture of Global and Local Models
• Local SGD Converges Fast and Communicates Little
• Federated Learning for Emoji Prediction in a Mobile Keyboard
• Distributed Mean Estimation with Limited Communication
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Federated Evaluation of On-device Personalization
• Variational Federated Multi-Task Learning
• Federated Learning Of Out-Of-Vocabulary Words
• Salvaging Federated Learning by Local Adaptation
• Private Federated Learning with Domain Adaptation
• Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization
• Generative Models for Effective ML on Private, Decentralized Datasets
• Federated User Representation Learning
• Federated Residual Learning

Cited by in corpus (76)

• Adaptive Personalized Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• Personalized Federated Learning with Moreau Envelopes
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• Federated Learning: A Signal Processing Perspective
• A Field Guide to Federated Optimization
• Federated Multi-Task Learning under a Mixture of Distributions
• The Non-IID Data Quagmire of Decentralized Machine Learning
• A review of federated learning in renewable energy applications: Potential, challenges, and future directions
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
• Exploiting Shared Representations for Personalized Federated Learning
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• Oort: Efficient Federated Learning via Guided Participant Selection
• FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks
• Hypernetwork-based Personalized Federated Learning for Multi-Institutional CT Imaging
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Personalized and privacy-preserving federated heterogeneous medical image analysis with PPPML-HMI
• Personalized Federated Learning using Hypernetworks
• Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
• Graph Federated Learning for CIoT Devices in Smart Home Applications
• Federated Learning Meets Multi-objective Optimization
• A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Things
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• Personalized Federated Learning with First Order Model Optimization
• Parameterized Knowledge Transfer for Personalized Federated Learning
• Federated Reconstruction: Partially Local Federated Learning
• An Optimal Transport Approach to Personalized Federated Learning
• Uncertainty Minimization for Personalized Federated Semi-Supervised Learning
• Federated brain tumor segmentation: an extensive benchmark
• Heterogeneity for the Win: One-Shot Federated Clustering
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Personalized Federated Learning through Local Memorization
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• Implicit Model Specialization through DAG-based Decentralized Federated Learning
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• Fast-adapting and Privacy-preserving Federated Recommender System
• Federated Multi-armed Bandits with Personalization
• A Theoretical Perspective on Differentially Private Federated Multi-task Learning
• Privacy-Preserving Distributed Learning for Residential Short-Term Load Forecasting
• Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
• You Can Backdoor Personalized Federated Learning
• Federated Transfer Learning with Task Personalization for Condition Monitoring in Ultrasonic Metal Welding
• Group privacy for personalized federated learning
• Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Clustered FedStack: Intermediate Global Models with Bayesian Information Criterion
• Personalized Federated Learning with Gaussian Processes
• Distributed Learning over Networks with Graph-Attention-Based Personalization
• Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
• Federated Learning for Commercial Image Sources
• Federated Mixture of Experts
• Personalized Federated Learning with Multi-branch Architecture
• Federated Learning with Intermediate Representation Regularization
• Federated Learning on Non-IID Data: A Survey
• Fairness-aware Agnostic Federated Learning
• Practical and Secure Federated Recommendation with Personalized Masks
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• Optimal Model Averaging: Towards Personalized Collaborative Learning
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization
• Private Multi-Task Learning: Formulation and Applications to Federated Learning
• Communication-Efficient Agnostic Federated Averaging
• Federated Learning for Open Banking
• Global Multiclass Classification and Dataset Construction via Heterogeneous Local Experts
• New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning
• QuPeL: Quantized Personalization with Applications to Federated Learning
• An Operator Splitting View of Federated Learning
• FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
• Controlled privacy leakage propagation throughout overlapping grouped learning
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks
• A Wasserstein Minimax Framework for Mixed Linear Regression
• Linear Speedup in Personalized Collaborative Learning
• One-Shot Clustering for Federated Learning