Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks

Abstract

Pre-training on graph neural networks (GNNs) aims to learn transferable knowledge for downstream tasks with unlabeled data, and it has recently become an active research area. The success of graph pre-training models is often attributed to the massive amount of input data. In this paper, however, we identify the curse of big data phenomenon in graph pre-training: more training data do not necessarily lead to better downstream performance. Motivated by this observation, we propose a better-with-less framework for graph pre-training: fewer, but carefully chosen data are fed into a GNN model to enhance pre-training. The proposed pre-training pipeline is called the data-active graph pre-training (APT) framework, and is composed of a graph selector and a pre-training model. The graph selector chooses the most representative and instructive data points based on the inherent properties of graphs as well as predictive uncertainty. The proposed predictive uncertainty, as feedback from the pre-training model, measures the confidence level of the model in the data. When fed with the chosen data, on the other hand, the pre-training model grasps an initial understanding of the new, unseen data, and at the same time attempts to remember the knowledge learned from previous data. Therefore, the integration and interaction between these two components form a unified framework (APT), in which graph pre-training is performed in a progressive and iterative way. Experiment results show that the proposed APT is able to obtain an efficient pre-training model with fewer training data and better downstream performance.

References in corpus (22)

• LINE: Large-scale Information Network Embedding
• Scaling Laws for Neural Language Models
• GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training
• graph2vec: Learning Distributed Representations of Graphs
• Deep Graph Contrastive Representation Learning
• Contrastive Multi-View Representation Learning on Graphs
• Random Walks: A Review of Algorithms and Applications
• Entropy Rate of Diffusion Processes on Complex Networks
• Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
• Local Similarity-Aware Deep Feature Embedding
• Graph Contrastive Learning Automated
• GPT-GNN: Generative Pre-Training of Graph Neural Networks
• Transfer Learning via Contextual Invariants for One-to-Many Cross-Domain Recommendation
• When Does Self-Supervision Help Graph Convolutional Networks?
• Fracking Deep Convolutional Image Descriptors
• G-Mixup: Graph Data Augmentation for Graph Classification
• Active Learning for Graph Embedding
• Exploring the Limits of Large Scale Pre-training
• GraphCL: Contrastive Self-Supervised Learning of Graph Representations
• Pre-Training Graph Neural Networks for Generic Structural Feature Extraction
• Using Similarity Measures to Select Pretraining Data for NER
• A Simple Framework for Uncertainty in Contrastive Learning