FastGL: A GPU-Efficient Framework for Accelerating Sampling-Based GNN Training at Large Scale

Abstract

Graph Neural Networks (GNNs) have shown great superiority on non-Euclidean graph data, achieving ground-breaking performance on various graph-related tasks. As a practical solution to train GNN on large graphs with billions of nodes and edges, the sampling-based training is widely adopted by existing training frameworks. However, through an in-depth analysis, we observe that the efficiency of existing sampling-based training frameworks is still limited due to the key bottlenecks lying in all three phases of sampling-based training, i.e., subgraph sample, memory IO, and computation. To this end, we propose FastGL, a GPU-efficient Framework for accelerating sampling-based training of GNN at Large scale by simultaneously optimizing all above three phases, taking into account both GPU characteristics and graph structure. Specifically, by exploiting the inherent overlap within graph structures, FastGL develops the Match-Reorder strategy to reduce the data traffic, which accelerates the memory IO without incurring any GPU memory overhead. Additionally, FastGL leverages a Memory-Aware computation method, harnessing the GPU memory's hierarchical nature to mitigate irregular data access during computation. FastGL further incorporates the Fused-Map approach aimed at diminishing the synchronization overhead during sampling. Extensive experiments demonstrate that FastGL can achieve an average speedup of 11.8x, 2.2x and 1.5x over the state-of-the-art frameworks PyG, DGL, and GNNLab, respectively.Our code is available at https://github.com/a1bc2def6g/fastgl-ae.

Accepted by ASPLOS 2024 fall cycle after major revision

References in corpus (14)

• DeepWalk: Online Learning of Social Representations
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• Scaling Graph Neural Networks with Approximate PageRank
• PyTorch-BigGraph: A Large-scale Graph Embedding System
• Stochastic Training of Graph Convolutional Networks with Variance Reduction
• C-SAW: A Framework for Graph Sampling and Random Walk on GPUs
• IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Research
• Cooperative Behavior Planning for Automated Driving using Graph Neural Networks
• Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining
• BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing
• BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling
• PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling