The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
arXiv:2305.13854 · doi:10.1145/3597428
Abstract
Graph Neural Networks (GNNs) are an emerging research field. This specialized Deep Neural Network (DNN) architecture is capable of processing graph structured data and bridges the gap between graph processing and Deep Learning (DL). As graphs are everywhere, GNNs can be applied to various domains including recommendation systems, computer vision, natural language processing, biology and chemistry. With the rapid growing size of real world graphs, the need for efficient and scalable GNN training solutions has come. Consequently, many works proposing GNN systems have emerged throughout the past few years. However, there is an acute lack of overview, categorization and comparison of such systems. We aim to fill this gap by summarizing and categorizing important methods and techniques for large-scale GNN solutions. In addition, we establish connections between GNN systems, graph processing systems and DL systems.
Accepted at ACM Computing Surveys
References in corpus (24)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Semi-Supervised Classification with Graph Convolutional Networks
- Near linear time algorithm to detect community structures in large-scale networks
- Language Models are Few-Shot Learners
- Fast Graph Representation Learning with PyTorch Geometric
- Revisiting Distributed Synchronous SGD
- Predicting multicellular function through multi-layer tissue networks
- Towards Deeper Graph Neural Networks
- FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling
- DeeperGCN: All You Need to Train Deeper GCNs
- Distributed GraphLab: A Framework for Machine Learning in the Cloud
- Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks
- How to scale distributed deep learning?
- Hybrid Edge Partitioner: Partitioning Large Power-Law Graphs under Memory Constraints
- GRIP: A Graph Neural Network Accelerator Architecture
- BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing
- Learned Low Precision Graph Neural Networks
- Learning Cross-Domain Representation with Multi-Graph Neural Network
- CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
- VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization
- Pregelix: Big(ger) Graph Analytics on A Dataflow Engine
- DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks
- ZIPPER: Exploiting Tile- and Operator-level Parallelism for General and Scalable Graph Neural Network Acceleration
- 2PS: High-Quality Edge Partitioning with Two-Phase Streaming