Ripple Walk Training: A Subgraph-based training framework for Large and Deep Graph Neural Network

Abstract

Graph neural networks (GNNs) have achieved outstanding performance in learning graph-structured data and various tasks. However, many current GNNs suffer from three common problems when facing large-size graphs or using a deeper structure: neighbors explosion, node dependence, and oversmoothing. Such problems attribute to the data structures of the graph itself or the designing of the multi-layers GNNs framework, and can lead to low training efficiency and high space complexity. To deal with these problems, in this paper, we propose a general subgraph-based training framework, namely Ripple Walk Training (RWT), for deep and large graph neural networks. RWT samples subgraphs from the full graph to constitute a mini-batch, and the full GNN is updated based on the mini-batch gradient. We analyze the high-quality subgraphs to train GNNs in a theoretical way. A novel sampling method Ripple Walk Sampler works for sampling these high-quality subgraphs to constitute the mini-batch, which considers both the randomness and connectivity of the graph-structured data. Extensive experiments on different sizes of graphs demonstrate the effectiveness and efficiency of RWT in training various GNNs (GCN & GAT).

References in corpus (11)

• Deep Convolutional Networks on Graph-Structured Data
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling
• GraphSAINT: Graph Sampling Based Inductive Learning Method
• MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing
• PairNorm: Tackling Oversmoothing in GNNs
• LanczosNet: Multi-Scale Deep Graph Convolutional Networks
• GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation
• Graph Star Net for Generalized Multi-Task Learning
• Fake News Detection on News-Oriented Heterogeneous Information Networks through Hierarchical Graph Attention

Cited by in corpus (6)

• Scalable Graph Neural Network Training: The Case for Sampling
• Subgraph Neural Networks
• Sampling methods for efficient training of graph convolutional networks: A survey
• CGCL: Collaborative Graph Contrastive Learning without Handcrafted Graph Data Augmentations
• Modeling Multi-Destination Trips with Sketch-Based Model
• T-EMDE: Sketching-based global similarity for cross-modal retrieval