Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation

Abstract

Consistency training is a popular method to improve deep learning models in computer vision and natural language processing. Graph neural networks (GNNs) have achieved remarkable performance in a variety of network science learning tasks, but to date no work has studied the effect of consistency training on large-scale graph problems. GNNs scale to large graphs by minibatch training and subsample node neighbors to deal with high degree nodes. We utilize the randomness inherent in the subsampling of neighbors and introduce a novel consistency training method to improve accuracy. For a target node we generate different neighborhood expansions, and distill the knowledge of the average of the predictions to the GNN. Our method approximates the expected prediction of the possible neighborhood samples and practically only requires a few samples. We demonstrate that our training method outperforms standard GNN training in several different settings, and yields the largest gains when label rates are low.

References in corpus (15)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Bootstrap your own latent: A new approach to self-supervised Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Combining Label Propagation and Simple Models Out-performs Graph Neural Networks
• When Does Self-Supervision Help Graph Convolutional Networks?
• Scalable Graph Neural Networks for Heterogeneous Graphs
• On Self-Distilling Graph Neural Network