GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training

Abstract

Graph representation learning has emerged as a powerful technique for addressing real-world problems. Various downstream graph learning tasks have benefited from its recent developments, such as node classification, similarity search, and graph classification. However, prior arts on graph representation learning focus on domain specific problems and train a dedicated model for each graph dataset, which is usually non-transferable to out-of-domain data. Inspired by the recent advances in pre-training from natural language processing and computer vision, we design Graph Contrastive Coding (GCC) -- a self-supervised graph neural network pre-training framework -- to capture the universal network topological properties across multiple networks. We design GCC's pre-training task as subgraph instance discrimination in and across networks and leverage contrastive learning to empower graph neural networks to learn the intrinsic and transferable structural representations. We conduct extensive experiments on three graph learning tasks and ten graph datasets. The results show that GCC pre-trained on a collection of diverse datasets can achieve competitive or better performance to its task-specific and trained-from-scratch counterparts. This suggests that the pre-training and fine-tuning paradigm presents great potential for graph representation learning.

11 pages, 5 figures, to appear in KDD 2020 proceedings

References in corpus (10)

• Modularity and community structure in networks
• DeepWalk: Online Learning of Social Representations
• LINE: Large-scale Information Network Embedding
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• struc2vec: Learning Node Representations from Structural Identity
• Vertex similarity in networks
• Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec
• DeepInf: Social Influence Prediction with Deep Learning
• Learning Structural Node Embeddings Via Diffusion Wavelets
• NetSMF: Large-Scale Network Embedding as Sparse Matrix Factorization

Cited by in corpus (77)

• Self-supervised Graph Learning for Recommendation
• Graph Contrastive Learning with Adaptive Augmentation
• Self-supervised Learning: Generative or Contrastive
• Contrastive Representation Learning: A Framework and Review
• Graph Self-Supervised Learning: A Survey
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning
• Hypergraph Contrastive Collaborative Filtering
• A Comprehensive Survey on Deep Graph Representation Learning
• Contrastive Meta Learning with Behavior Multiplicity for Recommendation
• Graph Neural Networks: Taxonomy, Advances and Trends
• Pre-training of Graph Neural Network for Modeling Effects of Mutations on Protein-Protein Binding Affinity
• Double-Scale Self-Supervised Hypergraph Learning for Group Recommendation
• TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning
• Graph Neural Networks for Financial Fraud Detection: A Review
• CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation
• A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainability
• SelfKG: Self-Supervised Entity Alignment in Knowledge Graphs
• Unnoticeable Backdoor Attacks on Graph Neural Networks
• On Representation Knowledge Distillation for Graph Neural Networks
• Hyper Meta-Path Contrastive Learning for Multi-Behavior Recommendation
• GOOD-D: On Unsupervised Graph Out-Of-Distribution Detection
• Dual Space Graph Contrastive Learning
• Self-supervised Contrastive Attributed Graph Clustering
• A Survey on Hypergraph Neural Networks: An In-Depth and Step-By-Step Guide
• Graph Communal Contrastive Learning
• Neighborhood-Enhanced Supervised Contrastive Learning for Collaborative Filtering
• CCGL: Contrastive Cascade Graph Learning
• Unsupervised Graph Poisoning Attack via Contrastive Loss Back-propagation
• Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentations
• Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries
• Company-as-Tribe: Company Financial Risk Assessment on Tribe-Style Graph with Hierarchical Graph Neural Networks
• Improving Generalizability of Graph Anomaly Detection Models via Data Augmentation
• Heterogeneous Graph Contrastive Learning with Meta-path Contexts and Adaptively Weighted Negative Samples
• Generating Counterfactual Hard Negative Samples for Graph Contrastive Learning
• Graph Foundation Models: Concepts, Opportunities and Challenges
• Exploring Edge Disentanglement for Node Classification
• Automated 3D Pre-Training for Molecular Property Prediction
• X-GOAL: Multiplex Heterogeneous Graph Prototypical Contrastive Learning
• Geometry-enhanced Pre-training on Interatomic Potentials
• A Self-supervised Riemannian GNN with Time Varying Curvature for Temporal Graph Learning
• GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning
• Geometer: Graph Few-Shot Class-Incremental Learning via Prototype Representation
• A Shapelet-based Framework for Unsupervised Multivariate Time Series Representation Learning
• Efficient Link Prediction via GNN Layers Induced by Negative Sampling
• GSQAS: Graph Self-supervised Quantum Architecture Search
• RPT: Toward Transferable Model on Heterogeneous Researcher Data via Pre-Training
• Predicting the Silent Majority on Graphs: Knowledge Transferable Graph Neural Network
• Pre-Training and Prompting for Few-Shot Node Classification on Text-Attributed Graphs
• Source Free Unsupervised Graph Domain Adaptation
• SketchNE: Embedding Billion-Scale Networks Accurately in One Hour
• Multi-View Subgraph Neural Networks: Self-Supervised Learning with Scarce Labeled Data
• Heterogeneous Social Event Detection via Hyperbolic Graph Representations
• CVTGAD: Simplified Transformer with Cross-View Attention for Unsupervised Graph-level Anomaly Detection
• Learning Robust Representation through Graph Adversarial Contrastive Learning
• Identifying Influential Brokers on Social Media from Social Network Structure
• A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective
• Position-Aware Subgraph Neural Networks with Data-Efficient Learning
• Self-supervised Representation Learning on Electronic Health Records with Graph Kernel Infomax
• Unify Graph Learning with Text: Unleashing LLM Potentials for Session Search
• Voucher Abuse Detection with Prompt-based Fine-tuning on Graph Neural Networks
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
• Self-Supervised Graph Representation Learning via Global Context Prediction
• Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks
• Towards Multimodal Graph Large Language Model
• Adaptive Multi-layer Contrastive Graph Neural Networks
• Handling Feature Heterogeneity with Learnable Graph Patches
• Boosting Multitask Learning on Graphs through Higher-Order Task Affinities
• InfoNCE is a Free Lunch for Semantically guided Graph Contrastive Learning
• EAGLE: Contrastive Learning for Efficient Graph Anomaly Detection
• Negative as Positive: Enhancing Out-of-distribution Generalization for Graph Contrastive Learning
• PreGIP: Watermarking the Pretraining of Graph Neural Networks for Deep Intellectual Property Protection
• Self-supervised Graph Representation Learning for Black Market Account Detection
• SPARK: Adaptive Low-Rank Knowledge Graph Modeling in Hybrid Geometric Spaces for Recommendation
• Learning Feature Encoder with Synthetic Anomalies for Weakly Supervised Graph Anomaly Detection
• Transfer Learning for Node Regression Applied to Spreading Prediction
• Tight PAC-Bayesian Risk Certificates for Contrastive Learning
• RUEL: Retrieval-Augmented User Representation with Edge Browser Logs for Sequential Recommendation