Iterative Graph Self-Distillation

Abstract

Recently, there has been increasing interest in the challenge of how to discriminatively vectorize graphs. To address this, we propose a method called Iterative Graph Self-Distillation (IGSD) which learns graph-level representation in an unsupervised manner through instance discrimination using a self-supervised contrastive learning approach. IGSD involves a teacher-student distillation process that uses graph diffusion augmentations and constructs the teacher model using an exponential moving average of the student model. The intuition behind IGSD is to predict the teacher network representation of the graph pairs under different augmented views. As a natural extension, we also apply IGSD to semi-supervised scenarios by jointly regularizing the network with both supervised and self-supervised contrastive loss. Finally, we show that finetuning the IGSD-trained models with self-training can further improve the graph representation power. Empirically, we achieve significant and consistent performance gain on various graph datasets in both unsupervised and semi-supervised settings, which well validates the superiority of IGSD.

arXiv commentThe Workshop on Self-Supervised Learning for the Web

References in corpus (19)

• Distilling the Knowledge in a Neural Network
• Finding and evaluating community structure in networks
• mixup: Beyond Empirical Risk Minimization
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Learning deep representations by mutual information estimation and maximization
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• What Makes for Good Views for Contrastive Learning?
• graph2vec: Learning Distributed Representations of Graphs
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Contrastive Multi-View Representation Learning on Graphs
• On Mutual Information Maximization for Representation Learning
• Order Matters: Sequence to sequence for sets
• Strategies for Pre-training Graph Neural Networks
• InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
• Graph Barlow Twins: A self-supervised representation learning framework for graphs
• The Multiscale Laplacian Graph Kernel
• Self-Distillation Amplifies Regularization in Hilbert Space
• Convolutional Kernel Networks for Graph-Structured Data

Cited by in corpus (5)

• Graph Self-Supervised Learning: A Survey
• On Representation Knowledge Distillation for Graph Neural Networks
• Graph Communal Contrastive Learning
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• Graph Neural Networks: Methods, Applications, and Opportunities