Decentralized Federated Learning: A Segmented Gossip Approach

Abstract

The emerging concern about data privacy and security has motivated the proposal of federated learning, which allows nodes to only synchronize the locally-trained models instead their own original data. Conventional federated learning architecture, inherited from the parameter server design, relies on highly centralized topologies and the assumption of large nodes-to-server bandwidths. However, in real-world federated learning scenarios the network capacities between nodes are highly uniformly distributed and smaller than that in a datacenter. It is of great challenges for conventional federated learning approaches to efficiently utilize network capacities between nodes. In this paper, we propose a model segment level decentralized federated learning to tackle this problem. In particular, we propose a segmented gossip approach, which not only makes full utilization of node-to-node bandwidth, but also has good training convergence. The experimental results show that even the training time can be highly reduced as compared to centralized federated learning.

arXiv commentAccepted to the 1st International Workshop on Federated Machine Learning for User Privacy and Data Confidentiality (FML'19)

References in corpus (7)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Towards Federated Learning at Scale: System Design
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• A Reliable Effective Terascale Linear Learning System
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• Gossip training for deep learning

Cited by in corpus (19)

• A Blockchain-based Decentralized Federated Learning Framework with Committee Consensus
• Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges
• Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networks
• A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guarantee
• Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Things
• GFL: A Decentralized Federated Learning Framework Based On Blockchain
• Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients
• Federated Learning for Smart Healthcare: A Survey
• Implicit Model Specialization through DAG-based Decentralized Federated Learning
• Mobility-Aware Decentralized Federated Learning with Joint Optimization of Local Iteration and Leader Selection for Vehicular Networks
• p2pGNN: A Decentralized Graph Neural Network for Node Classification in Peer-to-Peer Networks
• FedSAUC: A Similarity-Aware Update Control for Communication-Efficient Federated Learning in Edge Computing
• Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent
• Crossover-SGD: A gossip-based communication in distributed deep learning for alleviating large mini-batch problem and enhancing scalability
• The Effect of Training Parameters and Mechanisms on Decentralized Federated Learning based on MNIST Dataset
• Bottleneck Time Minimization for Distributed Iterative Processes: Speeding Up Gossip-Based Federated Learning on Networked Computers
• HADFL: Heterogeneity-aware Decentralized Federated Learning Framework