SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) are the first choice methods for graph machine learning problems thanks to their ability to learn state-of-the-art level representations from graph-structured data. However, centralizing a massive amount of real-world graph data for GNN training is prohibitive due to user-side privacy concerns, regulation restrictions, and commercial competition. Federated Learning is the de-facto standard for collaborative training of machine learning models over many distributed edge devices without the need for centralization. Nevertheless, training graph neural networks in a federated setting is vaguely defined and brings statistical and systems challenges. This work proposes SpreadGNN, a novel multi-task federated training framework capable of operating in the presence of partial labels and absence of a central server for the first time in the literature. SpreadGNN extends federated multi-task learning to realistic serverless settings for GNNs, and utilizes a novel optimization algorithm with a convergence guarantee, Decentralized Periodic Averaging SGD (DPA-SGD), to solve decentralized multi-task learning problems. We empirically demonstrate the efficacy of our framework on a variety of non-I.I.D. distributed graph-level molecular property prediction datasets with partial labels. Our results show that SpreadGNN outperforms GNN models trained over a central server-dependent federated learning system, even in constrained topologies. The source code is publicly available at https://github.com/FedML-AI/SpreadGNN

arXiv commentThree co-1st authors have equal contribution (alphabetical order)

References in corpus (24)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Federated Learning: Strategies for Improving Communication Efficiency
• Molecular Graph Convolutions: Moving Beyond Fingerprints
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Convolutional Networks on Graphs for Learning Molecular Fingerprints
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• SchNet: A continuous-filter convolutional neural network for modeling quantum interactions
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Don't Use Large Mini-Batches, Use Local SGD
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• A Convex Formulation for Learning Task Relationships in Multi-Task Learning
• Adaptive Federated Optimization
• FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks
• CoCoA: A General Framework for Communication-Efficient Distributed Optimization
• GraphFL: A Federated Learning Framework for Semi-Supervised Node Classification on Graphs
• Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD
• Federated Dynamic GNN with Secure Aggregation
• Towards Federated Graph Learning for Collaborative Financial Crimes Detection
• Toward Understanding the Impact of Staleness in Distributed Machine Learning
• Analysis of Distributed Stochastic Dual Coordinate Ascent

Cited by in corpus (4)

• Trustworthy Graph Neural Networks: Aspects, Methods and Trends
• FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks
• A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainability
• Personalized and privacy-preserving federated heterogeneous medical image analysis with PPPML-HMI