An Empirical Study of Graph Contrastive Learning

Abstract

Graph Contrastive Learning (GCL) establishes a new paradigm for learning graph representations without human annotations. Although remarkable progress has been witnessed recently, the success behind GCL is still left somewhat mysterious. In this work, we first identify several critical design considerations within a general GCL paradigm, including augmentation functions, contrasting modes, contrastive objectives, and negative mining techniques. Then, to understand the interplay of different GCL components, we conduct extensive, controlled experiments over a set of benchmark tasks on datasets across various domains. Our empirical studies suggest a set of general receipts for effective GCL, e.g., simple topology augmentations that produce sparse graph views bring promising performance improvements; contrasting modes should be aligned with the granularities of end tasks. In addition, to foster future research and ease the implementation of GCL algorithms, we develop an easy-to-use library PyGCL, featuring modularized CL components, standardized evaluation, and experiment management. We envision this work to provide useful empirical evidence of effective GCL algorithms and offer several insights for future research.

arXiv commentAccepted to NeurIPS 2021 (Datasets and Benchmarks track). Open-sourced library at https://github.com/GraphCL/PyGCL

References in corpus (26)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• The information bottleneck method
• Learning deep representations by mutual information estimation and maximization
• Fast Graph Representation Learning with PyTorch Geometric
• Graph Contrastive Learning with Adaptive Augmentation
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Pitfalls of Graph Neural Network Evaluation
• Graph Self-Supervised Learning: A Survey
• Exploring Simple Siamese Representation Learning
• Deep Graph Contrastive Representation Learning
• TUDataset: A collection of benchmark datasets for learning with graphs
• Strategies for Pre-training Graph Neural Networks
• On Variational Bounds of Mutual Information
• OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs
• Graph Barlow Twins: A self-supervised representation learning framework for graphs
• Are all negatives created equal in contrastive instance discrimination?
• Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks
• Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization
• Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks
• BYOL works even without batch statistics
• Motif-Driven Contrastive Learning of Graph Representations
• A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
• Self-supervised Graph Neural Networks without explicit negative sampling
• Towards Domain-Agnostic Contrastive Learning
• When Contrastive Learning Meets Active Learning: A Novel Graph Active Learning Paradigm with Self-Supervision