Salvaging Federated Learning by Local Adaptation

Abstract

Federated learning (FL) is a heavily promoted approach for training ML models on sensitive data, e.g., text typed by users on their smartphones. FL is expressly designed for training on data that are unbalanced and non-iid across the participants. To ensure privacy and integrity of the fedeated model, latest FL approaches use differential privacy or robust aggregation. We look at FL from the \emph{local} viewpoint of an individual participant and ask: (1) do participants have an incentive to participate in FL? (2) how can participants \emph{individually} improve the quality of their local models, without re-designing the FL framework and/or involving other participants? First, we show that on standard tasks such as next-word prediction, many participants gain no benefit from FL because the federated model is less accurate on their data than the models they can train locally on their own. Second, we show that differential privacy and robust aggregation make this problem worse by further destroying the accuracy of the federated model for many participants. Then, we evaluate three techniques for local adaptation of federated models: fine-tuning, multi-task learning, and knowledge distillation. We analyze where each is applicable and demonstrate that all participants benefit from local adaptation. Participants whose local models are poor obtain big accuracy improvements over conventional FL. Participants whose local models are better than the federated model\textemdash and who have no incentive to participate in FL today\textemdash improve less, but sufficiently to make the adapted federated model better than their local models.

arXiv commentupdate figures and correct typos

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning for Mobile Keyboard Prediction
• How To Backdoor Federated Learning
• Learning Differentially Private Recurrent Language Models
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• Personalized Federated Learning with Moreau Envelopes
• One-Shot Federated Learning
• Federated Evaluation of On-device Personalization
• Private Federated Learning with Domain Adaptation
• Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms

Cited by in corpus (34)

• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• A Field Guide to Federated Optimization
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Federated Mutual Learning
• Exploiting Shared Representations for Personalized Federated Learning
• Differentially Private Learning Needs Better Features (or Much More Data)
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks
• Parameterized Knowledge Transfer for Personalized Federated Learning
• Federated Reconstruction: Partially Local Federated Learning
• Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
• Optimality and Stability in Federated Learning: A Game-theoretic Approach
• What Do We Mean by Generalization in Federated Learning?
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• DistFL: Distribution-aware Federated Learning for Mobile Scenarios
• Federated Learning on Non-IID Data: A Survey
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• IFedAvg: Interpretable Data-Interoperability for Federated Learning
• FedHealth 2: Weighted Federated Transfer Learning via Batch Normalization for Personalized Healthcare
• Privacy Assessment of Federated Learning using Private Personalized Layers
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization
• Federated Multi-Target Domain Adaptation
• Private Multi-Task Learning: Formulation and Applications to Federated Learning
• Federated Learning for Open Banking
• More Industry-friendly: Federated Learning with High Efficient Design
• New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning
• Federated Few-Shot Learning with Adversarial Learning
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
• On Data Efficiency of Meta-learning
• Linear Speedup in Personalized Collaborative Learning