Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation

Abstract

Federated learning (FL), which utilizes communication between the server (core) and local devices (edges) to indirectly learn from more data, is an emerging field in deep learning research. Recently, Knowledge Distillation-based FL methods with notable performance and high applicability have been suggested. In this paper, we choose knowledge distillation-based FL method as our baseline and tackle a challenging problem that ensues from using these methods. Especially, we focus on the problem incurred in the server model that tries to mimic different datasets, each of which is unique to an individual edge device. We dub the problem 'edge bias', which occurs when multiple teacher models trained on different datasets are used individually to distill knowledge. We introduce this nuisance that occurs in certain scenarios of FL, and to alleviate it, we propose a simple yet effective distillation scheme named 'buffered distillation'. In addition, we also experimentally show that this scheme is effective in mitigating the straggler problem caused by delayed edges.

arXiv comment10 pages

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Federated Machine Learning: Concept and Applications
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• FedMD: Heterogenous Federated Learning via Model Distillation
• One-Shot Federated Learning
• IBM Federated Learning: an Enterprise Framework White Paper V0.1
• Federated Learning with Matched Averaging
• Paraphrasing Complex Network: Network Compression via Factor Transfer
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning

Cited by in corpus (2)

• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer