Federated Meta-Learning with Fast Convergence and Efficient Communication

Abstract

Statistical and systematic challenges in collaboratively training machine learning models across distributed networks of mobile devices have been the bottlenecks in the real-world application of federated learning. In this work, we show that meta-learning is a natural choice to handle these issues, and propose a federated meta-learning framework FedMeta, where a parameterized algorithm (or meta-learner) is shared, instead of a global model in previous approaches. We conduct an extensive empirical evaluation on LEAF datasets and a real-world production dataset, and demonstrate that FedMeta achieves a reduction in required communication cost by 2.82-4.33 times with faster convergence, and an increase in accuracy by 3.23%-14.84% as compared to Federated Averaging (FedAvg) which is a leading optimization algorithm in federated learning. Moreover, FedMeta preserves user privacy since only the parameterized algorithm is transmitted between mobile devices and central servers, and no raw data is collected onto the servers.

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Towards Federated Learning at Scale: System Design
• Matching Networks for One Shot Learning
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• On First-Order Meta-Learning Algorithms
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Federated Optimization in Heterogeneous Networks
• Learning to reinforcement learn
• Fair Resource Allocation in Federated Learning
• LEAF: A Benchmark for Federated Settings
• Don't Use Large Mini-Batches, Use Local SGD
• A Closer Look at Few-shot Classification
• Deep Meta-Learning: Learning to Learn in the Concept Space
• Deep & Cross Network for Ad Click Predictions

Cited by in corpus (71)

• Federated Machine Learning: Concept and Applications
• FedMD: Heterogenous Federated Learning via Model Distillation
• Personalized Federated Learning: A Meta-Learning Approach
• Three Approaches for Personalization with Applications to Federated Learning
• LEAF: A Benchmark for Federated Settings
• Ditto: Fair and Robust Federated Learning Through Personalization
• A Field Guide to Federated Optimization
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• Federated Forest
• Edge Intelligence: Architectures, Challenges, and Applications
• Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
• Exploiting Shared Representations for Personalized Federated Learning
• Federated Learning for Healthcare Informatics
• Federated Multi-view Matrix Factorization for Personalized Recommendations
• Federated Graph Classification over Non-IID Graphs
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare
• K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning
• Inexact-ADMM Based Federated Meta-Learning for Fast and Continual Edge Learning
• FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximation
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• Federated Meta Learning Enhanced Acoustic Radio Cooperative Framework for Ocean of Things Underwater Acoustic Communications
• GraphFL: A Federated Learning Framework for Semi-Supervised Node Classification on Graphs
• Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
• Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs
• Dif-MAML: Decentralized Multi-Agent Meta-Learning
• A Privacy Preserving System for Movie Recommendations Using Federated Learning
• Federated Reconstruction: Partially Local Federated Learning
• A Systematic Literature Review on Federated Learning: From A Model Quality Perspective
• Uncertainty Minimization for Personalized Federated Semi-Supervised Learning
• FedEval: A Holistic Evaluation Framework for Federated Learning
• Fast-adapting and Privacy-preserving Federated Recommender System
• MAMO: Memory-Augmented Meta-Optimization for Cold-start Recommendation
• Dynamic backdoor attacks against federated learning
• FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
• A Blockchain-based Reliable Federated Meta-learning for Metaverse: A Dual Game Framework
• What Do We Mean by Generalization in Federated Learning?
• Federated Composite Optimization
• FedGL: Federated Graph Learning Framework with Global Self-Supervision
• Applications of Federated Learning in Smart Cities: Recent Advances, Taxonomy, and Open Challenges
• Meta-Learning surrogate models for sequential decision making
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• FedSiam: Towards Adaptive Federated Semi-Supervised Learning
• Federated Mixture of Experts
• Federated Learning on Non-IID Data: A Survey
• FedE: Embedding Knowledge Graphs in Federated Setting
• Benchmarking Differential Privacy and Federated Learning for BERT Models
• Towards Practical Few-shot Federated NLP
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• Privacy-preserving Federated Primal-dual Learning for Non-convex and Non-smooth Problems with Model Sparsification
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• MetaSelector: Meta-Learning for Recommendation with User-Level Adaptive Model Selection
• Learning to Recommend via Meta Parameter Partition
• FedHome: Cloud-Edge based Personalized Federated Learning for In-Home Health Monitoring
• Federated Extra-Trees with Privacy Preserving
• Robustness of Meta Matrix Factorization Against Strict Privacy Constraints
• Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
• Meta Matrix Factorization for Federated Rating Predictions
• Personalized Adaptive Meta Learning for Cold-start User Preference Prediction
• Demystifying the Effects of Non-Independence in Federated Learning
• Federated Few-Shot Learning with Adversarial Learning
• On Data Efficiency of Meta-learning
• PrivNet: Safeguarding Private Attributes in Transfer Learning for Recommendation
• Management of Resource at the Network Edge for Federated Learning
• Zero Shot on the Cold-Start Problem: Model-Agnostic Interest Learning for Recommender Systems
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection
• CatFedAvg: Optimising Communication-efficiency and Classification Accuracy in Federated Learning
• Federated Marginal Personalization for ASR Rescoring
• Compositional federated learning: Applications in distributionally robust averaging and meta learning