From Canonical Correlation Analysis to Self-supervised Graph Neural Networks

Abstract

We introduce a conceptually simple yet effective model for self-supervised representation learning with graph data. It follows the previous methods that generate two views of an input graph through data augmentation. However, unlike contrastive methods that focus on instance-level discrimination, we optimize an innovative feature-level objective inspired by classical Canonical Correlation Analysis. Compared with other works, our approach requires none of the parameterized mutual information estimator, additional projector, asymmetric structures, and most importantly, negative samples which can be costly. We show that the new objective essentially 1) aims at discarding augmentation-variant information by learning invariant representations, and 2) can prevent degenerated solutions by decorrelating features in different dimensions. Our theoretical analysis further provides an understanding for the new objective which can be equivalently seen as an instantiation of the Information Bottleneck Principle under the self-supervised setting. Despite its simplicity, our method performs competitively on seven public graph datasets. The code is available at: https://github.com/hengruizhang98/CCA-SSG.

arXiv commentAccepted by NeurIPS 2021 main conference

References in corpus (14)

• The information bottleneck method
• Learning deep representations by mutual information estimation and maximization
• Graph Contrastive Learning with Adaptive Augmentation
• Variational Graph Auto-Encoders
• Exploring Simple Siamese Representation Learning
• Deep Graph Contrastive Representation Learning
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Learning Robust Representations via Multi-View Information Bottleneck
• Whitening for Self-Supervised Representation Learning
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Understanding Self-supervised Learning with Dual Deep Networks
• Iterative Graph Self-Distillation
• On Feature Decorrelation in Self-Supervised Learning

Cited by in corpus (3)

• Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement
• Counterfactual Learning on Graphs: A Survey
• CARL-G: Clustering-Accelerated Representation Learning on Graphs