Federated Mutual Learning

Abstract

Federated learning (FL) enables collaboratively training deep learning models on decentralized data. However, there are three types of heterogeneities in FL setting bringing about distinctive challenges to the canonical federated learning algorithm (FedAvg). First, due to the Non-IIDness of data, the global shared model may perform worse than local models that solely trained on their private data; Second, the objective of center server and clients may be different, where center server seeks for a generalized model whereas client pursue a personalized model, and clients may run different tasks; Third, clients may need to design their customized model for various scenes and tasks; In this work, we present a novel federated learning paradigm, named Federated Mutual Leaning (FML), dealing with the three heterogeneities. FML allows clients training a generalized model collaboratively and a personalized model independently, and designing their private customized models. Thus, the Non-IIDness of data is no longer a bug but a feature that clients can be personally served better. The experiments show that FML can achieve better performance than alternatives in typical FL setting, and clients can be benefited from FML with different models and tasks.

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• FedMD: Heterogenous Federated Learning via Model Distillation
• Federated Optimization in Heterogeneous Networks
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Salvaging Federated Learning by Local Adaptation
• HHHFL: Hierarchical Heterogeneous Horizontal Federated Learning for Electroencephalography
• Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
• Communication-Efficient Local Decentralized SGD Methods
• Adaptive Gradient-Based Meta-Learning Methods
• Knowledge Distillation for Multi-task Learning

Cited by in corpus (9)

• Adaptive Personalized Federated Learning
• A Field Guide to Federated Optimization
• Federated Learning for Computationally-Constrained Heterogeneous Devices: A Survey
• FedHe: Heterogeneous Models and Communication-Efficient Federated Learning
• FedH2L: Federated Learning with Model and Statistical Heterogeneity
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• Decentralised Person Re-Identification with Selective Knowledge Aggregation
• Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learning
• New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning