Self-supervised Learning on Graphs: Deep Insights and New Direction

Abstract

The success of deep learning notoriously requires larger amounts of costly annotated data. This has led to the development of self-supervised learning (SSL) that aims to alleviate this limitation by creating domain specific pretext tasks on unlabeled data. Simultaneously, there are increasing interests in generalizing deep learning to the graph domain in the form of graph neural networks (GNNs). GNNs can naturally utilize unlabeled nodes through the simple neighborhood aggregation that is unable to thoroughly make use of unlabeled nodes. Thus, we seek to harness SSL for GNNs to fully exploit the unlabeled data. Different from data instances in the image and text domains, nodes in graphs present unique structure information and they are inherently linked indicating not independent and identically distributed (or i.i.d.). Such complexity is a double-edged sword for SSL on graphs. On the one hand, it determines that it is challenging to adopt solutions from the image and text domains to graphs and dedicated efforts are desired. On the other hand, it provides rich information that enables us to build SSL from a variety of perspectives. Thus, in this paper, we first deepen our understandings on when, why, and which strategies of SSL work with GNNs by empirically studying numerous basic SSL pretext tasks on graphs. Inspired by deep insights from the empirical studies, we propose a new direction SelfTask to build advanced pretext tasks that are able to achieve state-of-the-art performance on various real-world datasets. The specific experimental settings to reproduce our results can be found in \url{https://github.com/ChandlerBang/SelfTask-GNN}.

References in corpus (7)

• Semi-Supervised Classification with Graph Convolutional Networks
• Distributed Representations of Sentences and Documents
• Inductive Representation Learning on Large Graphs
• Spectral Networks and Locally Connected Networks on Graphs
• Graph Neural Networks: A Review of Methods and Applications
• Simplifying Graph Convolutional Networks
• Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels

Cited by in corpus (32)

• Graph Contrastive Learning with Augmentations
• Graph Self-Supervised Learning: A Survey
• A Comprehensive Survey on Deep Graph Representation Learning
• FairDrop: Biased Edge Dropout for Enhancing Fairness in Graph Representation Learning
• Improving Fairness in Graph Neural Networks via Mitigating Sensitive Attribute Leakage
• DIG: A Turnkey Library for Diving into Graph Deep Learning Research
• Graph-MLP: Node Classification without Message Passing in Graph
• Self-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendation
• Graph Neural Networks: Methods, Applications, and Opportunities
• Exploring Edge Disentanglement for Node Classification
• SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks
• Automated Self-Supervised Learning for Graphs
• A Network Science perspective of Graph Convolutional Networks: A survey
• Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data
• Node Similarity Preserving Graph Convolutional Networks
• GraphMixup: Improving Class-Imbalanced Node Classification on Graphs by Self-supervised Context Prediction
• Socially-Aware Self-Supervised Tri-Training for Recommendation
• Self-Supervised Graph Co-Training for Session-based Recommendation
• From Local Structures to Size Generalization in Graph Neural Networks
• Jointly Attacking Graph Neural Network and its Explanations
• Hop-Count Based Self-Supervised Anomaly Detection on Attributed Networks
• Tree Decomposed Graph Neural Network
• Self-supervised Auxiliary Learning for Graph Neural Networks via Meta-Learning
• Hypergraph Pre-training with Graph Neural Networks
• Hyperbolic Hypergraphs for Sequential Recommendation
• CGCL: Collaborative Graph Contrastive Learning without Handcrafted Graph Data Augmentations
• Towards Self-Explainable Graph Neural Network
• Self-learn to Explain Siamese Networks Robustly
• Network representation learning: A macro and micro view
• Graph Feature Gating Networks
• Curvature Graph Neural Network
• Adaptive Transfer Learning on Graph Neural Networks