Adaptive Personalized Federated Learning

Abstract

Investigation of the degree of personalization in federated learning algorithms has shown that only maximizing the performance of the global model will confine the capacity of the local models to personalize. In this paper, we advocate an adaptive personalized federated learning (APFL) algorithm, where each client will train their local models while contributing to the global model. We derive the generalization bound of mixture of local and global models, and find the optimal mixing parameter. We also propose a communication-efficient optimization method to collaboratively learn the personalized models and analyze its convergence in both smooth strongly convex and nonconvex settings. The extensive experiments demonstrate the effectiveness of our personalization schema, as well as the correctness of established generalization theories.

[v3] Added convergence analysis for nonconvex losses and additional experiments along with new baselines [v2] A new generalization analysis is provided. Also, additional experiments are added

References in corpus (20)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Robust Aggregation for Federated Learning
• Federated Learning with Personalization Layers
• FedMD: Heterogenous Federated Learning via Model Distillation
• Personalized Federated Learning: A Meta-Learning Approach
• Three Approaches for Personalization with Applications to Federated Learning
• Agnostic Federated Learning
• Federated Learning of a Mixture of Global and Local Models
• On the Convergence of Local Descent Methods in Federated Learning
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Federated Evaluation of On-device Personalization
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
• Federated Mutual Learning
• Salvaging Federated Learning by Local Adaptation
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Is Local SGD Better than Minibatch SGD?
• Semi-Cyclic Stochastic Gradient Descent
• On the Sample Complexity of Adversarial Multi-Source PAC Learning
• On Localized Discrepancy for Domain Adaptation

Cited by in corpus (82)

• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Personalized Federated Learning: A Meta-Learning Approach
• From Distributed Machine Learning to Federated Learning: A Survey
• Three Approaches for Personalization with Applications to Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• Personalized Federated Learning with Moreau Envelopes
• A Field Guide to Federated Optimization
• Federated Multi-Task Learning under a Mixture of Distributions
• An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning
• Concept drift detection and adaptation for federated and continual learning
• A review of federated learning in renewable energy applications: Potential, challenges, and future directions
• An Overview of Federated Learning at the Edge and Distributed Ledger Technologies for Robotic and Autonomous Systems
• An Efficient and Reliable Asynchronous Federated Learning Scheme for Smart Public Transportation
• Exploiting Shared Representations for Personalized Federated Learning
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling
• Hierarchical Personalized Federated Learning Over Massive Mobile Edge Computing Networks
• Differentiated Federated Reinforcement Learning Based Traffic Offloading on Space-Air-Ground Integrated Networks
• Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentation
• Personalized and privacy-preserving federated heterogeneous medical image analysis with PPPML-HMI
• Graph Federated Learning for CIoT Devices in Smart Home Applications
• Edge-assisted Democratized Learning Towards Federated Analytics
• Federated Continual Learning with Weighted Inter-client Transfer
• A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Things
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• Semi-Synchronous Personalized Federated Learning over Mobile Edge Networks
• Decentralized federated learning of deep neural networks on non-iid data
• Personalized Federated Learning with First Order Model Optimization
• Parameterized Knowledge Transfer for Personalized Federated Learning
• Federated Reconstruction: Partially Local Federated Learning
• An Optimal Transport Approach to Personalized Federated Learning
• Uncertainty Minimization for Personalized Federated Semi-Supervised Learning
• FedRFQ: Prototype-Based Federated Learning with Reduced Redundancy, Minimal Failure, and Enhanced Quality
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Personalized Federated Learning through Local Memorization
• SCEI: A Smart-Contract Driven Edge Intelligence Framework for IoT Systems
• Implicit Model Specialization through DAG-based Decentralized Federated Learning
• Fast-adapting and Privacy-preserving Federated Recommender System
• Specialized federated learning using a mixture of experts
• Federated Multi-armed Bandits with Personalization
• Continual Horizontal Federated Learning for Heterogeneous Data
• Personalized Graph Federated Learning with Differential Privacy
• Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence
• What Do We Mean by Generalization in Federated Learning?
• Federated Composite Optimization
• Distributed Learning over Networks with Graph-Attention-Based Personalization
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Personalized Federated Learning with Gaussian Processes
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• Federated Learning for Commercial Image Sources
• Federated Mixture of Experts
• Federated $f$-Differential Privacy
• Personalized Federated Learning with Multi-branch Architecture
• Federated Learning on Non-IID Data: A Survey
• Privacy-preserving patient clustering for personalized federated learning
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• Modular Federated Learning
• Resource-Constrained Federated Learning with Heterogeneous Labels and Models
• Optimal Model Averaging: Towards Personalized Collaborative Learning
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
• IFedAvg: Interpretable Data-Interoperability for Federated Learning
• Personalized Federated Learning of Driver Prediction Models for Autonomous Driving
• Private Multi-Task Learning: Formulation and Applications to Federated Learning
• Federated Learning with Heterogeneous Labels and Models for Mobile Activity Monitoring
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• Federated Learning for Open Banking
• QuPeL: Quantized Personalization with Applications to Federated Learning
• An Operator Splitting View of Federated Learning
• New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning
• Dynamic Attention-based Communication-Efficient Federated Learning
• More Industry-friendly: Federated Learning with High Efficient Design
• FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• WAFFLE: Weighted Averaging for Personalized Federated Learning
• Local Learning at the Network Edge for Efficient & Secure Real-Time Predictive Analytics
• Controlled privacy leakage propagation throughout overlapping grouped learning
• Towards Heterogeneous Clients with Elastic Federated Learning
• Linear Speedup in Personalized Collaborative Learning
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection