Strategies for Pre-training Graph Neural Networks

Abstract

Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effectively use pre-training on graph datasets. In this paper, we develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training on multiple graph classification datasets. We find that naive strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks. In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.

arXiv commentAccepted as a spotlight to ICLR 2020

References in corpus (13)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distributed Representations of Words and Phrases and their Compositionality
• Neural Message Passing for Quantum Chemistry
• Fast Graph Representation Learning with PyTorch Geometric
• Representation Learning on Graphs with Jumping Knowledge Networks
• Variational Graph Auto-Encoders
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Atom2Vec: learning atoms for materials discovery
• subgraph2vec: Learning Distributed Representations of Rooted Sub-graphs from Large Graphs
• Position-aware Graph Neural Networks
• Learning Graph-Level Representation for Drug Discovery
• The Weisfeiler-Lehman Method and Graph Isomorphism Testing
• Pre-training Graph Neural Networks with Kernels

Cited by in corpus (83)

• Graph Neural Networks: A Review of Methods and Applications
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Graph Self-Supervised Learning: A Survey
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detection
• Pre-training Molecular Graph Representation with 3D Geometry
• Graph Neural Networks: Taxonomy, Advances and Trends
• Few-Shot Graph Learning for Molecular Property Prediction
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Pre-training of Graph Neural Network for Modeling Effects of Mutations on Protein-Protein Binding Affinity
• Do Transformers Really Perform Bad for Graph Representation?
• OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs
• How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• A Survey of Label-noise Representation Learning: Past, Present and Future
• 3D Infomax improves GNNs for Molecular Property Prediction
• GPT-GNN: Generative Pre-Training of Graph Neural Networks
• Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method
• On Representation Knowledge Distillation for Graph Neural Networks
• Learning to Make Generalizable and Diverse Predictions for Retrosynthesis
• Graph Clustering with Graph Neural Networks
• CCGL: Contrastive Cascade Graph Learning
• Subgraph Neural Networks
• Motif-based Graph Self-Supervised Learning for Molecular Property Prediction
• Learning Gradient Fields for Molecular Conformation Generation
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning
• Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs
• Hierarchical Inter-Message Passing for Learning on Molecular Graphs
• Large-Scale Representation Learning on Graphs via Bootstrapping
• Heterogeneous Network Representation Learning: A Unified Framework with Survey and Benchmark
• MARS: Markov Molecular Sampling for Multi-objective Drug Discovery
• Interpreting and Unifying Graph Neural Networks with An Optimization Framework
• Can Graph Neural Networks Count Substructures?
• Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• Learn molecular representations from large-scale unlabeled molecules for drug discovery
• SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks
• Node Classification on Graphs with Few-Shot Novel Labels via Meta Transformed Network Embedding
• Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data
• Boost then Convolve: Gradient Boosting Meets Graph Neural Networks
• Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining
• Automated Self-Supervised Learning for Graphs
• Meta-Learning GNN Initializations for Low-Resource Molecular Property Prediction
• Edge Representation Learning with Hypergraphs
• Unsupervised Point Cloud Pre-Training via Occlusion Completion
• Motif-Driven Contrastive Learning of Graph Representations
• Dual-view Molecule Pre-training
• Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structures
• MIMOSA: Multi-constraint Molecule Sampling for Molecule Optimization
• Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs
• Iterative Graph Self-Distillation
• Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning
• Self-supervised Graph Neural Networks without explicit negative sampling
• Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence
• CoCoSum: Contextual Code Summarization with Multi-Relational Graph Neural Network
• Towards Domain-Agnostic Contrastive Learning
• GraphAttacker: A General Multi-Task GraphAttack Framework
• Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction
• Graph-Aware Transformer: Is Attention All Graphs Need?
• Adaptive-Step Graph Meta-Learner for Few-Shot Graph Classification
• Calibrating and Improving Graph Contrastive Learning
• Graph convolutions that can finally model local structure
• Weisfeiler and Lehman Go Cellular: CW Networks
• Pre-Training Graph Neural Networks for Cold-Start Users and Items Representation
• Pre-training Graph Transformer with Multimodal Side Information for Recommendation
• Wasserstein Embedding for Graph Learning
• Self-supervised Auxiliary Learning for Graph Neural Networks via Meta-Learning
• Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction
• An Empirical Study of Graph Contrastive Learning
• GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering
• PanRep: Graph neural networks for extracting universal node embeddings in heterogeneous graphs
• EGC2: Enhanced Graph Classification with Easy Graph Compression
• Weakly-supervised Text Classification Based on Keyword Graph
• Meta-Learning to Improve Pre-Training
• Pairwise Half-graph Discrimination: A Simple Graph-level Self-supervised Strategy for Pre-training Graph Neural Networks
• Universal Representation for Code
• Pre-Trained Models for Heterogeneous Information Networks
• Graph Context Encoder: Graph Feature Inpainting for Graph Generation and Self-supervised Pretraining
• Explicit Semantic Cross Feature Learning via Pre-trained Graph Neural Networks for CTR Prediction
• Realistic molecule optimization on a learned graph manifold
• Network representation learning: A macro and micro view
• Cross-lingual Semantic Role Labeling with Model Transfer