FedRAD: Federated Robust Adaptive Distillation

Abstract

The robustness of federated learning (FL) is vital for the distributed training of an accurate global model that is shared among large number of clients. The collaborative learning framework by typically aggregating model updates is vulnerable to model poisoning attacks from adversarial clients. Since the shared information between the global server and participants are only limited to model parameters, it is challenging to detect bad model updates. Moreover, real-world datasets are usually heterogeneous and not independent and identically distributed (Non-IID) among participants, which makes the design of such robust FL pipeline more difficult. In this work, we propose a novel robust aggregation method, Federated Robust Adaptive Distillation (FedRAD), to detect adversaries and robustly aggregate local models based on properties of the median statistic, and then performing an adapted version of ensemble Knowledge Distillation. We run extensive experiments to evaluate the proposed method against recently published works. The results show that FedRAD outperforms all other aggregators in the presence of adversaries, as well as in heterogeneous data distributions.

arXiv commentAccepted for 1st NeurIPS Workshop on New Frontiers in Federated Learning (NFFL 2021), Virtual Meeting

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Byzantine-Robust Federated Machine Learning through Adaptive Model Averaging
• Zero-Shot Knowledge Distillation in Deep Networks
• Federated Learning Meets Multi-objective Optimization
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Robust Aggregation for Adaptive Privacy Preserving Federated Learning in Healthcare