Improving Federated Learning Personalization via Model Agnostic Meta Learning

Abstract

Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data. A natural scenario arises with data created on mobile phones by the activity of their users. Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually. In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number of similarities with the objective of personalization for FL. We present FL as a natural source of practical applications for MAML algorithms, and make the following observations. 1) The popular FL algorithm, Federated Averaging, can be interpreted as a meta learning algorithm. 2) Careful fine-tuning can yield a global model with higher accuracy, which is at the same time easier to personalize. However, solely optimizing for the global model accuracy yields a weaker personalization result. 3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim. These results raise new questions for FL, MAML, and broader ML research.

References in corpus (13)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Federated Learning for Mobile Keyboard Prediction
• On the Convergence of FedAvg on Non-IID Data
• Learning Differentially Private Recurrent Language Models
• On First-Order Meta-Learning Algorithms
• Lookahead Optimizer: k steps forward, 1 step back
• LEAF: A Benchmark for Federated Settings
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• How to train your MAML
• Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD

Cited by in corpus (98)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Personalized Federated Learning: A Meta-Learning Approach
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• LEAF: A Benchmark for Federated Settings
• Ditto: Fair and Robust Federated Learning Through Personalization
• Personalized Federated Learning with Moreau Envelopes
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• A Field Guide to Federated Optimization
• Federated Multi-Task Learning under a Mixture of Distributions
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
• Federated Mutual Learning
• Exploiting Shared Representations for Personalized Federated Learning
• Salvaging Federated Learning by Local Adaptation
• Federated Learning for Healthcare Informatics
• FedLab: A Flexible Federated Learning Framework
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• Inexact-ADMM Based Federated Meta-Learning for Fast and Continual Edge Learning
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Personalized and privacy-preserving federated heterogeneous medical image analysis with PPPML-HMI
• Fairness and Accuracy in Federated Learning
• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Graph Federated Learning for CIoT Devices in Smart Home Applications
• Federated Continual Learning: Concepts, Challenges, and Solutions
• Dif-MAML: Decentralized Multi-Agent Meta-Learning
• A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Things
• FedCD: Improving Performance in non-IID Federated Learning
• Semi-Synchronous Personalized Federated Learning over Mobile Edge Networks
• Decentralized federated learning of deep neural networks on non-iid data
• Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge based Framework
• Personalized Federated Learning with First Order Model Optimization
• Parameterized Knowledge Transfer for Personalized Federated Learning
• Federated Reconstruction: Partially Local Federated Learning
• FedCM: Federated Learning with Client-level Momentum
• An Optimal Transport Approach to Personalized Federated Learning
• Dynamic Fusion based Federated Learning for COVID-19 Detection
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Personalized Federated Learning through Local Memorization
• FedEval: A Holistic Evaluation Framework for Federated Learning
• On the Outsized Importance of Learning Rates in Local Update Methods
• Federated Residual Learning
• Implicit Model Specialization through DAG-based Decentralized Federated Learning
• Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• Specialized federated learning using a mixture of experts
• Federated Multi-armed Bandits with Personalization
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Fast-adapting and Privacy-preserving Federated Recommender System
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis
• What Do We Mean by Generalization in Federated Learning?
• Turn Signal Prediction: A Federated Learning Case Study
• A Blockchain-based Reliable Federated Meta-learning for Metaverse: A Dual Game Framework
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence
• Distributed Learning over Networks with Graph-Attention-Based Personalization
• A Clustering Strategy for Enhanced FL-Based Intrusion Detection in IoT Networks
• Personalized Federated Learning with Gaussian Processes
• Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Federated Composite Optimization
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• Federated Mixture of Experts
• Federated Learning on Non-IID Data: A Survey
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
• Federated Learning with Intermediate Representation Regularization
• Personalized Federated Learning with Multi-branch Architecture
• Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning
• SoK: Training Machine Learning Models over Multiple Sources with Privacy Preservation
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• Personalised Federated Learning On Heterogeneous Feature Spaces
• Federated Block Coordinate Descent Scheme for Learning Global and Personalized Models
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• IFedAvg: Interpretable Data-Interoperability for Federated Learning
• PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization
• Trends and Advancements in Deep Neural Network Communication
• Personalised Federated Learning: A Combinational Approach
• Data Selection for Efficient Model Update in Federated Learning
• Communication-Efficient Agnostic Federated Averaging
• Exploiting Heterogeneity in Robust Federated Best-Arm Identification
• Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems
• New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning
• Robustness of Meta Matrix Factorization Against Strict Privacy Constraints
• Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
• FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
• An Expectation-Maximization Perspective on Federated Learning
• When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Network
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• Local Learning at the Network Edge for Efficient & Secure Real-Time Predictive Analytics
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• Controlled privacy leakage propagation throughout overlapping grouped learning
• On Data Efficiency of Meta-learning
• Enhancing WiFi Multiple Access Performance with Federated Deep Reinforcement Learning