GPT-GNN: Generative Pre-Training of Graph Neural Networks

Abstract

Graph neural networks (GNNs) have been demonstrated to be powerful in modeling graph-structured data. However, training GNNs usually requires abundant task-specific labeled data, which is often arduously expensive to obtain. One effective way to reduce the labeling effort is to pre-train an expressive GNN model on unlabeled data with self-supervision and then transfer the learned model to downstream tasks with only a few labels. In this paper, we present the GPT-GNN framework to initialize GNNs by generative pre-training. GPT-GNN introduces a self-supervised attributed graph generation task to pre-train a GNN so that it can capture the structural and semantic properties of the graph. We factorize the likelihood of the graph generation into two components: 1) Attribute Generation and 2) Edge Generation. By modeling both components, GPT-GNN captures the inherent dependency between node attributes and graph structure during the generative process. Comprehensive experiments on the billion-scale Open Academic Graph and Amazon recommendation data demonstrate that GPT-GNN significantly outperforms state-of-the-art GNN models without pre-training by up to 9.1% across various downstream tasks.

arXiv commentPublished on KDD 2020

References in corpus (14)

• Decoupled Weight Decay Regularization
• Semi-Supervised Classification with Graph Convolutional Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Inductive Representation Learning on Large Graphs
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Spectral Networks and Locally Connected Networks on Graphs
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Fast Graph Representation Learning with PyTorch Geometric
• Momentum Contrast for Unsupervised Visual Representation Learning
• Variational Graph Auto-Encoders
• Strategies for Pre-training Graph Neural Networks
• Efficient Graph Generation with Graph Recurrent Attention Networks
• Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks
• Heterogeneous Graph Transformer

Cited by in corpus (12)

• Graph Self-Supervised Learning: A Survey
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation
• Social Recommendation with Self-Supervised Metagraph Informax Network
• Augmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting
• FoodSAM: Any Food Segmentation
• Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries
• Pitfalls in Link Prediction with Graph Neural Networks: Understanding the Impact of Target-link Inclusion & Better Practices
• RPT: Toward Transferable Model on Heterogeneous Researcher Data via Pre-Training
• Source Free Unsupervised Graph Domain Adaptation
• Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems
• Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks