Ginex: SSD-enabled Billion-scale Graph Neural Network Training on a Single Machine via Provably Optimal In-memory Caching

Abstract

Recently, Graph Neural Networks (GNNs) have been receiving a spotlight as a powerful tool that can effectively serve various inference tasks on graph structured data. As the size of real-world graphs continues to scale, the GNN training system faces a scalability challenge. Distributed training is a popular approach to address this challenge by scaling out CPU nodes. However, not much attention has been paid to disk-based GNN training, which can scale up the single-node system in a more cost-effective manner by leveraging high-performance storage devices like NVMe SSDs. We observe that the data movement between the main memory and the disk is the primary bottleneck in the SSD-based training system, and that the conventional GNN training pipeline is sub-optimal without taking this overhead into account. Thus, we propose Ginex, the first SSD-based GNN training system that can process billion-scale graph datasets on a single machine. Inspired by the inspector-executor execution model in compiler optimization, Ginex restructures the GNN training pipeline by separating sample and gather stages. This separation enables Ginex to realize a provably optimal replacement algorithm, known as Belady's algorithm, for caching feature vectors in memory, which account for the dominant portion of I/O accesses. According to our evaluation with four billion-scale graph datasets, Ginex achieves 2.11x higher training throughput on average (up to 2.67x at maximum) than the SSD-extended PyTorch Geometric.

arXiv commentPublished in 2022 International Conference on Very Large Databases (VLDB)

References in corpus (10)

• Array Programming with NumPy
• A Comprehensive Survey on Graph Neural Networks
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• Stochastic Training of Graph Convolutional Networks with Variance Reduction
• PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest
• AGL: a Scalable System for Industrial-purpose Graph Machine Learning
• Scalable Realistic Recommendation Datasets through Fractal Expansions
• Marius: Learning Massive Graph Embeddings on a Single Machine
• Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Graphs

Cited by in corpus (4)

• Graph Neural Networks for Financial Fraud Detection: A Review
• Accelerating Sampling and Aggregation Operations in GNN Frameworks with GPU Initiated Direct Storage Accesses
• A Comprehensive Benchmark on Spectral GNNs: The Impact on Efficiency, Memory, and Effectiveness
• LPS-GNN : Deploying Graph Neural Networks on Graphs with 100-Billion Edges