Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training

Abstract

It is hard to directly implement Graph Neural Networks (GNNs) on large scaled graphs. Besides of existed neighbor sampling techniques, scalable methods decoupling graph convolutions and other learnable transformations into preprocessing and post classifier allow normal minibatch training. By replacing redundant concatenation operation with attention mechanism in SIGN, we propose Scalable and Adaptive Graph Neural Networks (SAGN). SAGN can adaptively gather neighborhood information among different hops. To further improve scalable models on semi-supervised learning tasks, we propose Self-Label-Enhance (SLE) framework combining self-training approach and label propagation in depth. We add base model with a scalable node label module. Then we iteratively train models and enhance train set in several stages. To generate input of node label module, we directly apply label propagation based on one-hot encoded label vectors without inner random masking. We find out that empirically the label leakage has been effectively alleviated after graph convolutions. The hard pseudo labels in enhanced train set participate in label propagation with true labels. Experiments on both inductive and transductive datasets demonstrate that, compared with other sampling-based and sampling-free methods, SAGN achieves better or comparable results and SLE can further improve performance.

arXiv comment23 pages, 13 figures, fixed typos, add authors

References in corpus (22)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Simplifying Graph Convolutional Networks
• Disease Prediction using Graph Convolutional Networks: Application to Autism Spectrum Disorder and Alzheimer's Disease
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling
• GraphSAINT: Graph Sampling Based Inductive Learning Method
• Link Prediction Based on Graph Neural Networks
• Adaptive Sampling Towards Fast Graph Representation Learning
• Fake News Detection on Social Media using Geometric Deep Learning
• SIGN: Scalable Inception Graph Neural Networks
• Combining Label Propagation and Simple Models Out-performs Graph Neural Networks
• Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification
• Unifying Graph Convolutional Neural Networks and Label Propagation
• Scalable Graph Neural Networks for Heterogeneous Graphs
• Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels
• Self-Enhanced GNN: Improving Graph Neural Networks Using Model Outputs

Cited by in corpus (4)

• Graph Attention Multi-Layer Perceptron
• Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction
• Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks
• Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs