The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
arXiv:2305.13854 · doi:10.1145/3597428
Abstract
Graph Neural Networks (GNNs) are an emerging research field. This specialized Deep Neural Network (DNN) architecture is capable of processing graph structured data and bridges the gap between graph processing and Deep Learning (DL). As graphs are everywhere, GNNs can be applied to various domains including recommendation systems, computer vision, natural language processing, biology and chemistry. With the rapid growing size of real world graphs, the need for efficient and scalable GNN training solutions has come. Consequently, many works proposing GNN systems have emerged throughout the past few years. However, there is an acute lack of overview, categorization and comparison of such systems. We aim to fill this gap by summarizing and categorizing important methods and techniques for large-scale GNN solutions. In addition, we establish connections between GNN systems, graph processing systems and DL systems.
arXiv commentAccepted at ACM Computing Surveys
References in corpus (58)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Community structure in social and biological networks
- A Comprehensive Survey on Graph Neural Networks
- Semi-Supervised Classification with Graph Convolutional Networks
- Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
- An overview of gradient descent optimization algorithms
- Near linear time algorithm to detect community structures in large-scale networks
- Language Models are Few-Shot Learners
- Graph Convolutional Neural Networks for Web-Scale Recommender Systems
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
- Fast Graph Representation Learning with PyTorch Geometric
- MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding
- Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks
- Revisiting Distributed Synchronous SGD
- Predicting multicellular function through multi-layer tissue networks
- Towards Deeper Graph Neural Networks
- Open Graph Benchmark: Datasets for Machine Learning on Graphs
- FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling
- Gunrock: A High-Performance Graph Processing Library on the GPU
- GraphSAINT: Graph Sampling Based Inductive Learning Method
- Representation Learning on Graphs: Methods and Applications
- Learning Multiagent Communication with Backpropagation
- DeeperGCN: All You Need to Train Deeper GCNs
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
- Residual Gated Graph ConvNets
- HyperGCN: A New Method of Training Graph Convolutional Networks on Hypergraphs
- APAN: Asynchronous Propagation Attention Network for Real-time Temporal Graph Embedding
- OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs
- Machine Learning on Graphs: A Model and Comprehensive Taxonomy
- Temporal Graph Networks for Deep Learning on Dynamic Graphs
- Distributed GraphLab: A Framework for Machine Learning in the Cloud
- Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
- Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks
- Communication optimization strategies for distributed deep neural network training: A survey
- Accelerating Large Scale Real-Time GNN Inference using Channel Pruning
- How to scale distributed deep learning?
- High-Level Programming Abstractions for Distributed Graph Processing
- Scalable Graph Neural Network Training: The Case for Sampling
- AGL: a Scalable System for Industrial-purpose Graph Machine Learning
- AutoGraph: Automated Graph Neural Network
- Hybrid Edge Partitioner: Partitioning Large Power-Law Graphs under Memory Constraints
- An analysis of the graph processing landscape
- GRIP: A Graph Neural Network Accelerator Architecture
- BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing
- Degree-Quant: Quantization-Aware Training for Graph Neural Networks
- Learned Low Precision Graph Neural Networks
- Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis
- Learning Cross-Domain Representation with Multi-Graph Neural Network
- CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
- VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization
- Pregelix: Big(ger) Graph Analytics on A Dataflow Engine
- Efficient Graph Neural Network Inference at Large Scale
- GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategy
- DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks
- ZIPPER: Exploiting Tile- and Operator-level Parallelism for General and Scalable Graph Neural Network Acceleration
- 2PS: High-Quality Edge Partitioning with Two-Phase Streaming
- Distributed Edge Partitioning for Trillion-edge Graphs