Federated Learning of a Mixture of Global and Local Models

Abstract

We propose a new optimization formulation for training federated learning models. The standard formulation has the form of an empirical risk minimization problem constructed to find a single global model trained from the private data stored across all participating devices. In contrast, our formulation seeks an explicit trade-off between this traditional global model and the local models, which can be learned by each device from its own private data without any communication. Further, we develop several efficient variants of SGD (with and without partial participation and with and without variance reduction) for solving the new formulation and prove communication complexity guarantees. Notably, our methods are similar but not identical to federated averaging / local SGD, thus shedding some light on the role of local steps in federated learning. In particular, we are the first to i) show that local steps can improve communication for problems with heterogeneous data, and ii) point out that personalization yields reduced communication complexity.

arXiv comment40 pages, 8 algorithms, 6 figures, 1 table (minor changes compared to the previous versions)

References in corpus (16)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Variational Federated Multi-Task Learning
• AIDE: Fast and Communication Efficient Distributed Optimization
• Variance Reduced Local SGD with Lower Communication Complexity
• First Analysis of Local GD on Heterogeneous Data
• Private Federated Learning with Domain Adaptation
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Semi-Cyclic Stochastic Gradient Descent
• Distributed Stochastic Multi-Task Learning with Graph Regularization
• L-SVRG and L-Katyusha with Arbitrary Sampling
• One Method to Rule Them All: Variance Reduction for Data, Parameters and Many New Methods
• A Stochastic Decoupling Method for Minimizing the Sum of Smooth and Non-Smooth Functions
• SAGA with Arbitrary Sampling

Cited by in corpus (58)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Personalized Federated Learning: A Meta-Learning Approach
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• Personalized Federated Learning with Moreau Envelopes
• FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data
• A Field Guide to Federated Optimization
• Optimal Client Sampling for Federated Learning
• Federated Multi-Task Learning under a Mixture of Distributions
• Exploiting Shared Representations for Personalized Federated Learning
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling
• Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharing
• Differentiated Federated Reinforcement Learning Based Traffic Offloading on Space-Air-Ground Integrated Networks
• Hypernetwork-based Personalized Federated Learning for Multi-Institutional CT Imaging
• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Personalized Federated Learning with First Order Model Optimization
• Parameterized Knowledge Transfer for Personalized Federated Learning
• Federated Reconstruction: Partially Local Federated Learning
• An Optimal Transport Approach to Personalized Federated Learning
• Personalized Federated Learning through Local Memorization
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
• Specialized federated learning using a mixture of experts
• Federated Multi-armed Bandits with Personalization
• What Do We Mean by Generalization in Federated Learning?
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Personalized Federated Learning with Gaussian Processes
• Distributed Learning over Networks with Graph-Attention-Based Personalization
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• Local SGD: Unified Theory and New Efficient Methods
• Federated $f$-Differential Privacy
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
• Error Compensated Distributed SGD Can Be Accelerated
• Federated Learning on Non-IID Data: A Survey
• Personalized Federated Learning with Multi-branch Architecture
• Analysis of regularized federated learning
• Privacy-preserving patient clustering for personalized federated learning
• Practical and Secure Federated Recommendation with Personalized Masks
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• Personalised Federated Learning On Heterogeneous Feature Spaces
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
• A Unified Analysis of Variational Inequality Methods: Variance Reduction, Sampling, Quantization and Coordinate Descent
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• Private Multi-Task Learning: Formulation and Applications to Federated Learning
• Federated Learning for Open Banking
• QuPeL: Quantized Personalization with Applications to Federated Learning
• More Industry-friendly: Federated Learning with High Efficient Design
• FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
• Multi-Frequency Federated Learning for Human Activity Recognition Using Head-Worn Sensors
• Local Learning at the Network Edge for Efficient & Secure Real-Time Predictive Analytics
• Demystifying the Effects of Non-Independence in Federated Learning
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection
• Optimization for Supervised Machine Learning: Randomized Algorithms for Data and Parameters
• Weight Divergence Driven Divide-and-Conquer Approach for Optimal Federated Learning from non-IID Data
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks