Federated Learning of a Mixture of Global and Local Models

Abstract

We propose a new optimization formulation for training federated learning models. The standard formulation has the form of an empirical risk minimization problem constructed to find a single global model trained from the private data stored across all participating devices. In contrast, our formulation seeks an explicit trade-off between this traditional global model and the local models, which can be learned by each device from its own private data without any communication. Further, we develop several efficient variants of SGD (with and without partial participation and with and without variance reduction) for solving the new formulation and prove communication complexity guarantees. Notably, our methods are similar but not identical to federated averaging / local SGD, thus shedding some light on the role of local steps in federated learning. In particular, we are the first to i) show that local steps can improve communication for problems with heterogeneous data, and ii) point out that personalization yields reduced communication complexity.

40 pages, 8 algorithms, 6 figures, 1 table (minor changes compared to the previous versions)

References in corpus (8)

• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• AIDE: Fast and Communication Efficient Distributed Optimization
• Variance Reduced Local SGD with Lower Communication Complexity
• Private Federated Learning with Domain Adaptation
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Semi-Cyclic Stochastic Gradient Descent
• L-SVRG and L-Katyusha with Arbitrary Sampling
• SAGA with Arbitrary Sampling

Cited by in corpus (57)

• Personalized Federated Learning: A Meta-Learning Approach
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• Personalized Federated Learning with Moreau Envelopes
• FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data
• A Field Guide to Federated Optimization
• Optimal Client Sampling for Federated Learning
• Federated Multi-Task Learning under a Mixture of Distributions
• Exploiting Shared Representations for Personalized Federated Learning
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling
• Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharing
• Differentiated Federated Reinforcement Learning Based Traffic Offloading on Space-Air-Ground Integrated Networks
• Hypernetwork-based Personalized Federated Learning for Multi-Institutional CT Imaging
• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Personalized Federated Learning with First Order Model Optimization
• Parameterized Knowledge Transfer for Personalized Federated Learning
• Federated Reconstruction: Partially Local Federated Learning
• An Optimal Transport Approach to Personalized Federated Learning
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Personalized Federated Learning through Local Memorization
• Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
• Specialized federated learning using a mixture of experts
• Federated Multi-armed Bandits with Personalization
• What Do We Mean by Generalization in Federated Learning?
• Distributed Learning over Networks with Graph-Attention-Based Personalization
• Personalized Federated Learning with Gaussian Processes
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Local SGD: Unified Theory and New Efficient Methods
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• Federated $f$-Differential Privacy
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
• Error Compensated Distributed SGD Can Be Accelerated
• Federated Learning on Non-IID Data: A Survey
• Personalized Federated Learning with Multi-branch Architecture
• Analysis of regularized federated learning
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• Privacy-preserving patient clustering for personalized federated learning
• Practical and Secure Federated Recommendation with Personalized Masks
• Personalised Federated Learning On Heterogeneous Feature Spaces
• A Unified Analysis of Variational Inequality Methods: Variance Reduction, Sampling, Quantization and Coordinate Descent
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• Private Multi-Task Learning: Formulation and Applications to Federated Learning
• Federated Learning for Open Banking
• More Industry-friendly: Federated Learning with High Efficient Design
• QuPeL: Quantized Personalization with Applications to Federated Learning
• FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
• Multi-Frequency Federated Learning for Human Activity Recognition Using Head-Worn Sensors
• Local Learning at the Network Edge for Efficient & Secure Real-Time Predictive Analytics
• Demystifying the Effects of Non-Independence in Federated Learning
• Weight Divergence Driven Divide-and-Conquer Approach for Optimal Federated Learning from non-IID Data
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection
• Optimization for Supervised Machine Learning: Randomized Algorithms for Data and Parameters