Scalable Graph Neural Network Training: The Case for Sampling

Abstract

Graph Neural Networks (GNNs) are a new and increasingly popular family of deep neural network architectures to perform learning on graphs. Training them efficiently is challenging due to the irregular nature of graph data. The problem becomes even more challenging when scaling to large graphs that exceed the capacity of single devices. Standard approaches to distributed DNN training, such as data and model parallelism, do not directly apply to GNNs. Instead, two different approaches have emerged in the literature: whole-graph and sample-based training. In this paper, we review and compare the two approaches. Scalability is challenging with both approaches, but we make a case that research should focus on sample-based training since it is a more promising approach. Finally, we review recent systems supporting sample-based training.

arXiv comment9 pages, 2 figures

References in corpus (16)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• A Comprehensive Survey on Graph Neural Networks
• TensorFlow: A system for large-scale machine learning
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Fast Graph Representation Learning with PyTorch Geometric
• One weird trick for parallelizing convolutional neural networks
• GraphSAINT: Graph Sampling Based Inductive Learning Method
• Adaptive Sampling Towards Fast Graph Representation Learning
• Stochastic Training of Graph Convolutional Networks with Variance Reduction
• Peregrine: A Pattern-Aware Graph Mining System
• Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks
• DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs
• Sampling methods for efficient training of graph convolutional networks: A survey
• Ripple Walk Training: A Subgraph-based training framework for Large and Deep Graph Neural Network
• Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networks

Cited by in corpus (4)

• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• GraphDistNet: A Graph-based Collision-distance Estimator for Gradient-based Trajectory Optimization
• Bingo: Radix-based Bias Factorization for Random Walk on Dynamic Graphs
• Hierarchical graph sampling based minibatch learning with chain preservation and variance reduction