Personalized Federated Learning: A Meta-Learning Approach

Abstract

In Federated Learning, we aim to train models across multiple computing units (users), while users can only communicate with a common central server, without exchanging their data samples. This mechanism exploits the computational power of all users and allows users to obtain a richer model as their models are trained over a larger set of data points. However, this scheme only develops a common output for all the users, and, therefore, it does not adapt the model to each user. This is an important missing feature, especially given the heterogeneity of the underlying data distribution for various users. In this paper, we study a personalized variant of the federated learning in which our goal is to find an initial shared model that current or new users can easily adapt to their local dataset by performing one or a few steps of gradient descent with respect to their own data. This approach keeps all the benefits of the federated learning architecture, and, by structure, leads to a more personalized model for each user. We show this problem can be studied within the Model-Agnostic Meta-Learning (MAML) framework. Inspired by this connection, we study a personalized variant of the well-known Federated Averaging algorithm and evaluate its performance in terms of gradient norm for non-convex loss functions. Further, we characterize how this performance is affected by the closeness of underlying distributions of user data, measured in terms of distribution distances such as Total Variation and 1-Wasserstein metric.

arXiv commentTo appear in 34th Conference on Neural Information Processing Systems (NeurIPS 2020)

References in corpus (24)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Learning Differentially Private Recurrent Language Models
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• On First-Order Meta-Learning Algorithms
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Fair Resource Allocation in Federated Learning
• Adaptive Personalized Federated Learning
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Federated Learning of a Mixture of Global and Local Models
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• Fast Context Adaptation via Meta-Learning
• On the Convergence of Local Descent Methods in Federated Learning
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• One-Shot Federated Learning
• Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
• Alpha MAML: Adaptive Model-Agnostic Meta-Learning
• Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning
• Differentially Private Meta-Learning

Cited by in corpus (80)

• FedML: A Research Library and Benchmark for Federated Machine Learning
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• Personalized Federated Learning with Moreau Envelopes
• Federated Social Recommendation with Graph Neural Network
• A Field Guide to Federated Optimization
• Explainable, Domain-Adaptive, and Federated Artificial Intelligence in Medicine
• Federated Multi-Task Learning under a Mixture of Distributions
• Federated Learning with Spiking Neural Networks
• Subgraph Federated Learning with Missing Neighbor Generation
• An Overview of Federated Learning at the Edge and Distributed Ledger Technologies for Robotic and Autonomous Systems
• Salvaging Federated Learning by Local Adaptation
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• Federated Graph Classification over Non-IID Graphs
• Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• Inexact-ADMM Based Federated Meta-Learning for Fast and Continual Edge Learning
• Federated Learning Meets Natural Language Processing: A Survey
• FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks
• Federated Transfer-Ordered-Personalized Learning for Driver Monitoring Application
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• Personalized and privacy-preserving federated heterogeneous medical image analysis with PPPML-HMI
• Personalized Federated Learning using Hypernetworks
• GraphFL: A Federated Learning Framework for Semi-Supervised Node Classification on Graphs
• Edge-assisted Democratized Learning Towards Federated Analytics
• Federated Continual Learning with Weighted Inter-client Transfer
• Federated Continual Learning: Concepts, Challenges, and Solutions
• Graph Federated Learning for CIoT Devices in Smart Home Applications
• Dif-MAML: Decentralized Multi-Agent Meta-Learning
• PEPPER: Empowering User-Centric Recommender Systems over Gossip Learning
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Things
• FedCD: Improving Performance in non-IID Federated Learning
• Personalized Federated Learning with First Order Model Optimization
• Federated Reconstruction: Partially Local Federated Learning
• On the Outsized Importance of Learning Rates in Local Update Methods
• Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients
• On-device Online Learning and Semantic Management of TinyML Systems
• Implicit Model Specialization through DAG-based Decentralized Federated Learning
• Specialized federated learning using a mixture of experts
• Federated Multi-armed Bandits with Personalization
• Fast-adapting and Privacy-preserving Federated Recommender System
• Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis
• Personalized Graph Federated Learning with Differential Privacy
• Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• MetaPhys: Few-Shot Adaptation for Non-Contact Physiological Measurement
• Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
• Federated Mixture of Experts
• Loss Tolerant Federated Learning
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
• Federated Learning with Intermediate Representation Regularization
• SoK: Training Machine Learning Models over Multiple Sources with Privacy Preservation
• Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• Privacy-preserving patient clustering for personalized federated learning
• Federated Block Coordinate Descent Scheme for Learning Global and Personalized Models
• IFedAvg: Interpretable Data-Interoperability for Federated Learning
• Modular Federated Learning
• Prospects of federated machine learning in fluid dynamics
• Optimal Model Averaging: Towards Personalized Collaborative Learning
• Personalized Federated Learning of Driver Prediction Models for Autonomous Driving
• State-of-the-art Techniques in Deep Edge Intelligence
• Global Multiclass Classification and Dataset Construction via Heterogeneous Local Experts
• What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data
• Exploiting Heterogeneity in Robust Federated Best-Arm Identification
• Personalized Federated Learning over non-IID Data for Indoor Localization
• New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning
• QuPeL: Quantized Personalization with Applications to Federated Learning
• An Expectation-Maximization Perspective on Federated Learning
• Demystifying the Effects of Non-Independence in Federated Learning
• Local Learning at the Network Edge for Efficient & Secure Real-Time Predictive Analytics
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks
• Management of Resource at the Network Edge for Federated Learning
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach
• Collaborative and Federated Black-box Optimization: A Bayesian Optimization Perspective
• Linear Speedup in Personalized Collaborative Learning