Decentralized Federated Learning via Mutual Knowledge Transfer

Abstract

In this paper, we investigate the problem of decentralized federated learning (DFL) in Internet of things (IoT) systems, where a number of IoT clients train models collectively for a common task without sharing their private training data in the absence of a central server. Most of the existing DFL schemes are composed of two alternating steps, i.e., model updating and model averaging. However, averaging model parameters directly to fuse different models at the local clients suffers from client-drift especially when the training data are heterogeneous across different clients. This leads to slow convergence and degraded learning performance. As a possible solution, we propose the decentralized federated earning via mutual knowledge transfer (Def-KT) algorithm where local clients fuse models by transferring their learnt knowledge to each other. Our experiments on the MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets reveal that the proposed Def-KT algorithm significantly outperforms the baseline DFL methods with model averaging, i.e., Combo and FullAvg, especially when the training data are not independent and identically distributed (non-IID) across different clients.

Published in IEEE Internet of Things Journal

References in corpus (5)

• Overcoming catastrophic forgetting in neural networks
• Federated Learning: Challenges, Methods, and Future Directions
• Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
• Semi-supervised Deep Reinforcement Learning in Support of IoT and Smart City Services
• Geographic Gossip: Efficient Averaging for Sensor Networks

Cited by in corpus (9)

• Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges
• Decentralized Federated Learning through Proxy Model Sharing
• Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching
• Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
• FedBoosting: Federated Learning with Gradient Protected Boosting for Text Recognition
• Collaborative Distributed Machine Learning
• Decentralized Personalization for Federated Medical Image Segmentation via Gossip Contrastive Mutual Learning
• FLAGS Framework for Comparative Analysis of Federated Learning Algorithms
• Nonparametric Decentralized Detection and Sparse Sensor Selection via Multi-Sensor Online Kernel Scalar Quantization