Pre-Training Graph Neural Networks for Generic Structural Feature Extraction

Abstract

Graph neural networks (GNNs) are shown to be successful in modeling applications with graph structures. However, training an accurate GNN model requires a large collection of labeled data and expressive features, which might be inaccessible for some applications. To tackle this problem, we propose a pre-training framework that captures generic graph structural information that is transferable across tasks. Our framework can leverage the following three tasks: 1) denoising link reconstruction, 2) centrality score ranking, and 3) cluster preserving. The pre-training procedure can be conducted purely on the synthetic graphs, and the pre-trained GNN is then adapted for downstream applications. With the proposed pre-training procedure, the generic structural information is learned and preserved, thus the pre-trained GNN requires less amount of labeled data and fewer domain-specific features to achieve high performance on different downstream tasks. Comprehensive experiments demonstrate that our proposed framework can significantly enhance the performance of various tasks at the level of node, link, and graph.

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distributed Representations of Words and Phrases and their Compositionality
• Distributed Representations of Sentences and Documents
• Inductive Representation Learning on Large Graphs
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Higher-order organization of complex networks
• Subgraph Centrality in Complex Networks
• Influence maximization in complex networks through optimal percolation
• Predicting multicellular function through multi-layer tissue networks

Cited by in corpus (7)

• Graph Self-Supervised Learning: A Survey
• Graph Neural Networks: Taxonomy, Advances and Trends
• Graph Neural Networks: Methods, Applications, and Opportunities
• Towards Robust Graph Contrastive Learning
• Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization
• Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks
• Self-supervised Auxiliary Learning for Graph Neural Networks via Meta-Learning