Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization

Abstract

Graph neural networks (GNNs) have achieved superior performance in various applications, but training dedicated GNNs can be costly for large-scale graphs. Some recent work started to study the pre-training of GNNs. However, none of them provide theoretical insights into the design of their frameworks, or clear requirements and guarantees towards their transferability. In this work, we establish a theoretically grounded and practically useful framework for the transfer learning of GNNs. Firstly, we propose a novel view towards the essential graph information and advocate the capturing of it as the goal of transferable GNN training, which motivates the design of EGI (Ego-Graph Information maximization) to analytically achieve this goal. Secondly, when node features are structure-relevant, we conduct an analysis of EGI transferability regarding the difference between the local graph Laplacians of the source and target graphs. We conduct controlled synthetic experiments to directly justify our theoretical conclusions. Comprehensive experiments on two real-world network datasets show consistent results in the analyzed setting of direct-transfering, while those on large-scale knowledge graphs show promising results in the more practical setting of transfering with fine-tuning.

arXiv commentNeurIPS 2021

References in corpus (15)

• Emergence of scaling in random networks
• Statistical mechanics of complex networks
• Distributed Representations of Words and Phrases and their Compositionality
• Learning deep representations by mutual information estimation and maximization
• struc2vec: Learning Node Representations from Structural Identity
• Variational Graph Auto-Encoders
• GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models
• A Survey on Graph Kernels
• Contrastive Multi-View Representation Learning on Graphs
• Strategies for Pre-training Graph Neural Networks
• Easing Embedding Learning by Comprehensive Transcription of Heterogeneous Information Networks
• Universal Invariant and Equivariant Graph Neural Networks
• Graph Kernels: A Survey
• Graph Normalizing Flows
• Pre-Training Graph Neural Networks for Generic Structural Feature Extraction

Cited by in corpus (9)

• Graph Self-Supervised Learning: A Survey
• A Survey on Influence Maximization: From an ML-Based Combinatorial Optimization
• A Review of Graph Neural Networks and Their Applications in Power Systems
• Collaborate to Adapt: Source-Free Graph Domain Adaptation via Bi-directional Adaptation
• Scalable Multitask Learning Using Gradient-based Estimation of Task Affinity
• Boosting Multitask Learning on Graphs through Higher-Order Task Affinities
• Handling Feature Heterogeneity with Learnable Graph Patches
• An Empirical Study of Graph Contrastive Learning
• LinkThief: Combining Generalized Structure Knowledge with Node Similarity for Link Stealing Attack against GNN