When Does Self-Supervision Help Graph Convolutional Networks?

Abstract

Self-supervision as an emerging technique has been employed to train convolutional neural networks (CNNs) for more transferrable, generalizable, and robust representation learning of images. Its introduction to graph convolutional networks (GCNs) operating on graph data is however rarely explored. In this study, we report the first systematic exploration and assessment of incorporating self-supervision into GCNs. We first elaborate three mechanisms to incorporate self-supervision into GCNs, analyze the limitations of pretraining & finetuning and self-training, and proceed to focus on multi-task learning. Moreover, we propose to investigate three novel self-supervised learning tasks for GCNs with theoretical rationales and numerical comparisons. Lastly, we further integrate multi-task self-supervision into graph adversarial training. Our results show that, with properly designed task forms and incorporation mechanisms, self-supervision benefits GCNs in gaining more generalizability and robustness. Our codes are available at https://github.com/Shen-Lab/SS-GCNs.

Supplementary materials are available at https://yyou1996.github.io/files/icml2020_ssgcn_supplement.pdf. ICML 2020

References in corpus (20)

• Explaining and Harnessing Adversarial Examples
• Semi-Supervised Classification with Graph Convolutional Networks
• The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains
• Adversarial Attacks on Neural Networks for Graph Data
• Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
• How Powerful are Graph Neural Networks?
• Link Prediction Based on Graph Neural Networks
• Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
• Adversarial Attack on Graph Structured Data
• Adversarial Examples on Graph Data: Deep Insights into Attack and Defense
• Selfie: Self-supervised Pretraining for Image Embedding
• Estimating the Trace of the Matrix Inverse by Interpolating from the Diagonal of an Approximate Inverse
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Semi-Supervised Learning -- A Statistical Physics Approach
• GraphDefense: Towards Robust Graph Convolutional Networks
• Adversarial Defense Framework for Graph Neural Network
• Bias-Variance Tradeoff of Graph Laplacian Regularizer
• Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels
• When Does Self-supervision Improve Few-shot Learning?
• L${}^2$-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networks

Cited by in corpus (21)

• Graph Contrastive Learning with Augmentations
• Graph Self-Supervised Learning: A Survey
• Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future
• Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detection
• Graph Attention Multi-Layer Perceptron
• CGC: Contrastive Graph Clustering for Community Detection and Tracking
• Augmentations in Hypergraph Contrastive Learning: Fabricated and Generative
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks
• ROD: Reception-aware Online Distillation for Sparse Graphs
• Automated Self-Supervised Learning for Graphs
• Node Similarity Preserving Graph Convolutional Networks
• From Local Structures to Size Generalization in Graph Neural Networks
• Pre-Training Graph Neural Networks for Cold-Start Users and Items Representation
• Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks
• Leveraging Meta-path Contexts for Classification in Heterogeneous Information Networks
• Cross-Modality Protein Embedding for Compound-Protein Affinity and Contact Prediction
• Graph Feature Gating Networks
• RRLFSOR: An Efficient Self-Supervised Learning Strategy of Graph Convolutional Networks
• Network representation learning: A macro and micro view
• Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation