Personalized Federated Learning with Moreau Envelopes

Abstract

Federated learning (FL) is a decentralized and privacy-preserving machine learning technique in which a group of clients collaborate with a server to learn a global model without sharing clients' data. One challenge associated with FL is statistical diversity among clients, which restricts the global model from delivering good performance on each client's task. To address this, we propose an algorithm for personalized FL (pFedMe) using Moreau envelopes as clients' regularized loss functions, which help decouple personalized model optimization from the global model learning in a bi-level problem stylized for personalized FL. Theoretically, we show that pFedMe's convergence rate is state-of-the-art: achieving quadratic speedup for strongly convex and sublinear speedup of order 2/3 for smooth nonconvex objectives. Experimentally, we verify that pFedMe excels at empirical performance compared with the vanilla FedAvg and Per-FedAvg, a meta-learning based personalized FL algorithm.

References in corpus (14)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Federated Learning with Personalization Layers
• FedMD: Heterogenous Federated Learning via Model Distillation
• Personalized Federated Learning: A Meta-Learning Approach
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Agnostic Federated Learning
• Federated Learning of a Mixture of Global and Local Models
• On the Convergence of Local Descent Methods in Federated Learning
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• One-Shot Federated Learning
• Adaptive Federated Optimization
• Federated Evaluation of On-device Personalization
• Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning

Cited by in corpus (33)

• FedALA: Adaptive Local Aggregation for Personalized Federated Learning
• From Distributed Machine Learning to Federated Learning: A Survey
• Ditto: Fair and Robust Federated Learning Through Personalization
• Topology-aware Federated Learning in Edge Computing: A Comprehensive Survey
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Salvaging Federated Learning by Local Adaptation
• Fairness and Accuracy in Federated Learning
• Personalized and privacy-preserving federated heterogeneous medical image analysis with PPPML-HMI
• Personalized Federated Learning using Hypernetworks
• Federated Reconstruction: Partially Local Federated Learning
• FedCM: Federated Learning with Client-level Momentum
• Model-Contrastive Federated Learning
• Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• You Can Backdoor Personalized Federated Learning
• Personalized Federated Continual Learning via Multi-granularity Prompt
• Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data
• Federated $f$-Differential Privacy
• Federated Learning with Intermediate Representation Regularization
• Practical and Secure Federated Recommendation with Personalized Masks
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• FedHealth 2: Weighted Federated Transfer Learning via Batch Normalization for Personalized Healthcare
• SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning
• Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor
• Blinder: End-to-end Privacy Protection in Sensing Systems via Personalized Federated Learning
• Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
• Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentation
• Federated Learning for Open Banking
• New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning
• UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection