Implicit Model Specialization through DAG-based Decentralized Federated Learning

Abstract

Federated learning allows a group of distributed clients to train a common machine learning model on private data. The exchange of model updates is managed either by a central entity or in a decentralized way, e.g. by a blockchain. However, the strong generalization across all clients makes these approaches unsuited for non-independent and identically distributed (non-IID) data. We propose a unified approach to decentralization and personalization in federated learning that is based on a directed acyclic graph (DAG) of model updates. Instead of training a single global model, clients specialize on their local data while using the model updates from other clients dependent on the similarity of their respective data. This specialization implicitly emerges from the DAG-based communication and selection of model updates. Thus, we enable the evolution of specialized models, which focus on a subset of the data and therefore cover non-IID data better than federated learning in a centralized or blockchain-based setup. To the best of our knowledge, the proposed solution is the first to unite personalization and poisoning robustness in fully decentralized federated learning. Our evaluation shows that the specialization of models emerges directly from the DAG-based communication of model updates on three different datasets. Furthermore, we show stable model accuracy and less variance across clients when compared to federated averaging.

to be published in Middleware '21

References in corpus (25)

• Fast unfolding of communities in large networks
• Finding and evaluating community structure in networks
• Modularity and community structure in networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• An Overview of Multi-Task Learning in Deep Neural Networks
• Federated Learning with Non-IID Data
• Federated Optimization in Heterogeneous Networks
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Clustered Multi-Task Learning: A Convex Formulation
• LEAF: A Benchmark for Federated Settings
• Ditto: Fair and Robust Federated Learning Through Personalization
• Gossip Learning with Linear Models on Fully Distributed Data
• Scaling Nakamoto Consensus to Thousands of Transactions per Second
• Decentralized Federated Learning: A Segmented Gossip Approach
• Federated Evaluation of On-device Personalization
• Variational Federated Multi-Task Learning
• Decentralized Collaborative Learning of Personalized Models over Networks
• Blockchain-Federated-Learning and Deep Learning Models for COVID-19 detection using CT Imaging
• Biscotti: A Ledger for Private and Secure Peer-to-Peer Machine Learning
• Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning
• Self-organizing Democratized Learning: Towards Large-scale Distributed Learning Systems

Cited by in corpus (1)

• MultiConfederated Learning: Inclusive Non-IID Data handling with Decentralized Federated Learning